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© Multilens imaging apparatus with correction of misregistration. 



*Z © This invention relates to a multi-lens imaging apparatus and, more particularly, to a multi-lens imaging 
^ apparatus for obtaining a single high-fine image by synthesizing a plurality of images obtained by picking up an 

image of a common object using a plurality of imaging systems and to an apparatus which can provide an 
W image having an aspect ratio different from that of an image obtained by a single imaging apparatus, especially, 
^ a panoramic image, by synthesizing a plurality of images using a plurality of imaging systems. This invention 
CO has as its object to obtain a high-fine image by synthesizing images from a plurality of imaging systems and, 

more specifically, to correct mis-registration such as keystone distortions generated in the images. 
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BACKGROUND OF THE INVENTION 
Field of the Invention 

5 The present invention relates to a multi-lens imaging apparatus and, more particularly, to a multi-lens 

imaging apparatus which obtains a single high-fine image by synthesizing a plurality of images obtained by 
picking up a common object image using a plurality of imaging systems and to an apparatus which can 
provide a image having a different aspect ratio from that of a image obtained by a single imaging 
apparatus, for example, a panoramic image, by synthesizing a plurality of images obtained using a plurality 

70 of imaging optical systems. 

Related Background Art 

When an ultra high-fine image consisting of 4,000 x 4,000 pixels is picked up by an imaging system, 
75 and the picked-up image is displayed on an ultra high-fine monitor, problems associated with techniques for 
achieving a high density and a high sensitivity in the imaging system are posed. As a method of solving 
such problems in the imaging system, the principle of a multi-lens imaging apparatus for picking up an 
image of a common object using two imaging systems each having a small number of pixels, and obtaining 
a single high-fine image by synthesizing the two images obtained by the two imaging systems has been 
20 proposed. In the multi-lens imaging apparatus based on this principle, as shown in Fig. 1, right and left side 
imaging systems 110 R and 110 L are prepared, and pick up an image of an object 101 by shifting their 
sampling points by a 1/2 spatial phase pitch. A right side image l R obtained by the right side imaging 
system 110 R and a left side image l L obtained by the left side imaging system 1 1 0 L are synthesized by a 
microprocessor (to be referred to as a "CPU" hereinafter) 120, thereby obtaining a single high-fine output 
25 image Iout as compared to a image obtained when the image of the object 101 is picked up by a single 
imaging system. 

Fig. 2 is an explanatory view for explaining a basic arrangement of the right and left side imaging 
systems 110 R and 110 L shown in. Fig. 1. 

The left side imaging system 110 L is constituted by a left side imaging optical system 111 L and a left 

30 side image sensor 112 L , and the right side imaging system 110 R is constituted by a right side imaging 
optical system 111 R and a right side image sensor 11 2 R . The right and left side imaging optical systems 
111 R and 1 1 1 u have equivalent specifications, and each comprises a zoom lens. The right and left side 
image sensors 112 R and 1 1 2 L have equivalent specifications, and each comprises a camera tube such as a 
saticon or a solid-state imaging element such as a CCD. Optical axes L R and L L of the right and left imaging 

35 systems 110 R and 110 L cross almost at a point O on an object surface 102, and these systems are 
arranged at positions axially symmetrical about a normal O-O* to the object surface 102. When the angle 
formed between each of the optical axes L R and L L and the normal O-O' to the object surface 102 (to be 
referred to as an "inclination angle" hereinafter) is represented by 6, 26 is defined as the convergence 
angle. 

40 In the multi-lens imaging apparatus, when the object distance is changed, the convergence angle 20 is 
changed by rotating each of the right and left side imaging systems 110 R and 110 L about an X mark shown 
in Fig. 2 in accordance with the change in object distance, thus performing an imaging operation. 

However, in the above-mentioned multi-lens imaging apparatus, only the principle shown in Fig. 1 and 
the results of principle experiments for demonstrating the principle are presented, and no detailed 

45 arrangement of the multi-lens imaging apparatus is presented. Thus, a problem associated with how to 
arrange the detailed multi-lens imaging apparatus utilizing the above-mentioned principle remains unsolved. 
Since the optical axes L R and L L of the right and left side imaging systems 110 R and 1.1 0 L are inclined with 
respect to the object surface 102, the right and left side images l R and l L obtained by the right and left side 
image sensors 112 R and 112 L suffer from trapezoidal distortions (to be referred to as "keystone distortions" 

so hereinafter), as shown in Fig. 3. As a result, when the right and left side images l R and l L are merely 
synthesized, a high-fine image cannot be obtained. 

Since the optical systems are arranged to have a convergence angle therebetween in principle, mis- 
registration between images obtained from the two imaging operation systems may occur. Fig. 4A shows 
the outline of mis-registration in a multi-lens imaging system. The system shown in Fig. 4A includes 

55 imaging optical systems (lenses) 910 and 920, and image sensors 911 and 921. The imaging optical 
systems 910 and 920 have optical axes 912 and 922, respectively. When an object image is picked up 
while the optical axes 912 and 922 are inclined through 6 in the x-z plane with respect to a central axis O- 
0\ an arbitrary object point on an object surface is represented by P. At this time, if image points of the 
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image sensors 911 and 921 with respect to P are respectively represented by R' and L*, since R' * L\ mis- 
registration occurs, and two images with respect to the object point P are formed on a simply added 
synthesized image 930, as shown in Fig. 4B. As a result, a high-fine image cannot be provided. 

Conventionally, as a panoramic imaging apparatus using a plurality of imaging optical systems and 

5 designed for dynamic images, for example, Japanese Laid-Open Patent Application No. 63-8641 is known. 
In an apparatus of this type, in general, the angle formed between the imaging optical systems is set by 
mechanical adjustment, so that image fields contact each other. However, it is very difficult to strictly 
register two adjacent images, and the boundary line between the two images normally becomes conspicu- 
ous. For the same reason as described above, since it is difficult to re-focus objects having different object 

w distances or to perform zooming of these objects, the above-mentioned apparatus is limited to a cut 
photographing use in units of scenes. 

As a method of changing the aspect ratio of a frame of a television or video signal (e.g., conversion 
between 4 : 3 of the NTSC and 16 : 9 of the HD or ED2), a method of trimming upper and lower portions or 
right and left portions of the frame is known. In this method, since a picked-up image is partially used, when 

75 an image is picked by an NTSC camera using a 4 : 3 image sensor, and is output onto a 16 : 9 HD monitor, 
a total of 25% upper and lower images are lost in addition to the fact that the number of pixels is originally 
short, thus considerably deteriorating image quality. Conversely, when an image is picked up by a 16 : 9 
HD camera, and is output onto a 4 : 3 NTSC monitor, no problem associated with image quality is posed, 
but the horizontal field angle is decreased by 1/3. 

20 

SUMMARY OF THE INVENTION 

It is an object of the present invention to obtain a high-fine image by synthesizing images from a 
plurality of imaging systems and. more specifically, to correct mis-registration such as keystone distortions 

25 generated in the images. 

In order to achieve the above object, there is provided an apparatus comprising detection means for 
detecting a mis-registration amount between imaging systems, and means for controlling at least a 
mechanism system or an image process system, so that the mis-registration amount detected by the 
detection means becomes equal to or smaller than a predetermined value, whereby images obtained from 

30 the imaging systems are synthesized while being registered with each other. 

In order to achieve the above object, there is also provided an apparatus comprising means for 
detecting imaging conditions of a plurality of imaging systems, means for detecting position information of 
an object from image signals output from the imaging systems, and means for synthesizing images output 
from the imaging systems on the basis of the imaging conditions and the position information. 

35 According to the present invention, an image having a desired aspect ratio and almost free from image 
deterioration can be obtained, and a panoramic image in which a boundary line is not conspicuous can also 
be obtained. 

According to the present invention, when a plurality of images are synthesized, the mis-registration 
amount of overlapping images may be partially detected. 
40 According to the present invention, means for setting a convergence angle of imaging systems to be a 
predetermined value may be arranged so as to change the aspect ratio of an image. 

BRIEF DESCRIPTION OF THE DRAWINGS 

45 Fig. 1 is a view for explaining the principle of a multi-lens imaging apparatus for obtaining a single high- 
fine image by synthesizing two images obtained by picking up an image of a common object using two 
imaging systems; 

Fig. 2 is an explanatory view for explaining a basic arrangement of right and left side imaging systems 
shown in Fig. 1 ; 

so Fig. 3 is an explanatory view for explaining keystone distortions generated in images obtained by right 
and left side image sensors shown in Fig. 1; 

Figs. 4A and 4B are explanatory views of mis-registration occurring in the prior art; 

Figs. 5A and 5B show a multi-lens imaging apparatus according to an embodiment of the present 
invention, in which Fig. 5A is a perspective view showing the arrangement of right and left imaging 
55 systems, and Fig. 5B is a block diagram for explaining operations of keystone distortion correction signal 
generation portions; 

Fig. 6 is an explanatory view for explaining the principle of correcting keystone distortions in the 
keystone distortion correction signal generation portions shown in Fig. 5B; 
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Fig. 7 is an explanatory view for explaining the principle of correcting keystone distortions in the 
keystone distortion correction signal generation portions shown in Fig. 5B; 

Fig. 8 is an explanatory view for explaining the principle of correcting keystone distortions in the 
keystone distortion correction signal generation portions shown in Fig. 5B; 
5 Fig. 9 is an explanatory view for explaining operations of camera tube electron beam scanning locus 
control drivers shown in Fig. 5B; 

Fig. 10 is a perspective view showing the arrangement of right and left side imaging systems in a multi- 
lens imaging apparatus according to another embodiment of the present invention; 

Fig. 11 is a block diagram for explaining operations of correction conversion portions in the multi-lens 

70 imaging apparatus according to the embodiment shown in Fig. 10; 

Fig. 12 is a perspective view showing the arrangement of right and left side imaging systems in a multi- 
lens imaging apparatus according to still another embodiment of the present invention; 
Fig. 13 is a block diagram showing operations of correction optical systems shown in Fig. 12; 
Fig. 14 is an explanatory view for explaining the principle of correcting keystone distortions in the 

75 correction optical systems shown in Fig. 12; 

Fig. 15 is an explanatory view for explaining the principle of correcting keystone distortions in the 
correction optical systems shown in Fig. 12; 

Fig. 16 is an explanatory view for explaining the principle of correcting keystone distortions in the 
correction optical systems shown in Fig. 12; 
20 Figs. 17A and 17B show a multi-lens imaging apparatus according to still another embodiment of the 
present invention, in which Fig. 17A is a perspective view showing the arrangement of right and left side 
imaging systems, and Fig. 17B is a block diagram for explaining a correction method of keystone 
distortions; 

Fig. 18 is an explanatory view for explaining operations of a keystone distortion conversion portion and a 
25 subtraction process circuit shown in Fig. 17B; 

Fig. 19 is a graph showing the relationship between the image difference and the inclination angle 
{conversion yariable); 

Figs. 20A and 20B show a multi-lens imaging apparatus according to still another embodiment of the 
present invention, in which Fig. 20A is a perspective view showing the arrangement of right and left side 
30 imaging systems, and Fig. 20B is a block diagram for explaining a correction method of keystone 
distortions; 

Figs. 21 A to 21 C show a multi-lens imaging apparatus according to still another embodiment of the 
present invention, in which Fig. 21 A is a perspective view showing the arrangement of right and left side 
imaging systems, Fig. 21 B is a block diagram for explaining an operation of a control portion, and Fig. 
35 21 C is a block diagram for explaining an image synthesizing operation; 

Fig. 22 is an explanatory view for explaining a basic arrangement of imaging systems in the multi-lens 
imaging apparatus according to the embodiment shown in Figs. 21 A to 21 C; 

Fig. 23 is a view for explaining response characteristics of each convergence angle motor shown in Fig. 
21 A; 

40 Figs. 24A and 24B show a multi-lens imaging apparatus according to still another embodiment of the 
present invention, in which Fig. 24A is a perspective view showing the arrangement of right and left side 
imaging systems, and Fig. 24B is a block diagram for explaining an operation of a calculation control 
portion; 

Figs. 25A and 25B show a multi-lens imaging apparatus according to still another embodiment of the 
45 present invention, in which Fig. 25A is a perspective view showing the arrangement of right and left side 
imaging systems, and Fig. 25B is a block diagram for explaining an operation of a calculation control 
portion; 

Figs. 26A and 266 show a multi-lens imaging apparatus according to still another embodiment of the 
present invention, in which Fig. 26A is a perspective view showing the arrangement of right and left side 
so imaging systems, and Fig. 26B is a block diagram for explaining an operation of a calculation control 
portion; 

Fig. 27 is a block diagram for explaining an- operation of a correlation calculation process portion in a 
multi-lens imaging apparatus according to still another embodiment of the present invention; 
Fig. 28 is a block diagram for explaining an operation of a magnification correlation calculation process 
55 portion in a multi-lens imaging apparatus according to still another embodiment of the present invention; 

Fig. 29 is an explanatory view for explaining a method of shortening the correlation calculation time in 
the magnification correlation calculation process portion shown in Fig. 28; 
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Fig. 30 is an explanatory view for explaining another method of shortening the correlation calculation time 
in the magnification correlation calculation process portion shown in Fig. 28; 

Fig. 31 is a view showing a basic arrangement according to still another embodiment of the present 
invention; 

5 Fig. 32 is a perspective view of optical systems of the embodiment shown in Fig. 31; 

Fig. 33 is a diagram showing the flow of a signal process for calculating a mis-registration amount; 
Fig. 34 is a view for explaining a problem; 

Fig. 35 is a diagram showing the flow of a compensation signal process in the embodiment shown in Fig. 
31; 

io Fig. 36 is a diagram showing the flow of a image signal process in the embodiment shown in Fig. 31 ; 

Fig. 37 is a perspective view showing optical systems according to still another embodiment of the 
present invention; 

Fig. 38 is a diagram showing the flow of a compensation signal process in the embodiment shown in Fig. 
37; 

75 Fig. 39 is a perspective view showing optical systems according to still another embodiment of the 
present invention; 

Fig. 40 is a diagram showing the flow of a compensation signal process in the embodiment shown in Fig. 
39; 

Fig. 41 is a perspective view showing optical systems according to still another embodiment of the 
20 present invention; 

Fig. 42 is a diagram showing the flow of a compensation signal process in the embodiment shown in Fig. 
41; 

Fig. 43 is a perspective view showing optical systems according to still another embodiment of the 
present invention; 

25 Fig. 44 is a diagram showing the flow of a signal process for calculating a mis-registration amount; 
Fig. 45 is a perspective view for explaining an optical effect; 

Fig. 46 is an explanatory view for explaining a method of obtaining an image having a desired aspect 
ratio; 

Fig. 47 is an explanatory view of the aspect ratio; 
30 Fig. 48 is a view showing an optical basic arrangement; 

Fig. 49 is a view showing a basic arrangement according to still another embodiment of the present 
invention; 

Fig. 50 is a perspective view showing the detailed arrangement of the embodiment shown in Fig. 49; 
Fig. 51 is a block diagram showing relationships among functions in the embodiment shown in Fig. 49; 
35 Fig. 52 is an explanatory view associated with a method of calculating three-dimensional position 
information of an object; 

Fig. 53 is an explanatory view associated with a method of calculating two-dimensional position 
information of an object; 

Figs. 54A and 54B are views showing a correlation calculation process; 
40 Fig. 55 is a graph showing a correlation value; and 

Fig. 56 is an explanatory view of an image correction process. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

45 Figs. 5A and 5B show an arrangement of a multi-lens imaging apparatus according to an embodiment 
of the present invention. 

The multi-lens imaging apparatus of this embodiment obtains a single high-fine image by synthesizing 
two images obtained by picking up an image of a common object using two imaging systems (right and left 
side imaging systems 10 R and 10 L ) 

so The arrangement of the right and left side imaging systems 10 R and 10 L will be described in detail 
below with reference to Fig. 5A. 

The left side imaging system 10 L includes a left side imaging optical system 11 L (see Fig. 8), a left side 
image sensor 12 L comprising a camera tube, and a camera tube electron beam scanning locus control 
driver 31 L for controlling the locus of scanning of the electron beam emitted from the camera tube 

55 constituting the left side image sensor 12 L . The left side imaging optical system 11 L includes lens groups 
21 L to 24 L including a zoom lens group 22 L and a focus lens group 24 Lf a zoom motor 25 L as a driving 
system for driving the zoom lens group 22 L , a focus motor 26 L as a driving system for driving the focus lens 
group 24 L , a mechanism system (not shown) and a driving system (convergence angle motor 27 L ) for 
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integrally rotating the left side imaging optical system 11 L and the left side image sensor 12 L in a plane 
including an optical axis L Lt and a rotational angle detecting means 28 L for detecting the rotational angle of 
the convergence angle motor 27 L . As the rotational angle detecting means 28 L , for example, an externally 
attached member such as a rotary encoder may be used, or a means such as a pulse motor for detecting 

5 the rotational angle by the driving system itself according to a driving method may be used. The right side 
imaging system 10 R has the same arrangement as that of the left side imaging system 10 L . 

Assume that when the focus motor 26 L and the zoom motor 25 L of the left side imaging optical system 
11 L are independently controlled by signals from a focus encoder 42 L and a zoom encoder 41 L , and a focus 
motor 26 R and a zoom motor 25 R of a right side imaging optical system 11 R are independently controlled 

10 by signals from a focus encoder 42 R and a zoom encoder 41 Rt a focal length f of the left side imaging 
optical system 1 1 L always coincides with a focal length f of the right side imaging optical system 1 1 Rl and 
an image formation magnification 0 of the left side imaging optical system 11 L always coincides with an 
image formation magnification & of the right side imaging optical system 1 1 R . 

At this time, a keystone distortion correction signal generation portion 32 L shown at the left side in Rg. 

75 5B can calculate the focal length f of the left side imaging optical system 11 L on the basis of a signal from 
the zoom encoder 41 L , and a keystone distortion correction signal generation portion 32 R shown at the right 
side in Fig. 5B can calculate the focal length f of the right side imaging optical system 1 1 R on the basis of a 
signal from the zoom encoder 41 R . Also, since the keystone distortion correction signal generation portion 
32 L can calculate an object distance to the left side imaging optical system 11 L on the basis of a signal from 

20 the focus encoder 42 Ll it can calculate the image formation magnification & of the left side imaging optical 
system 11 L - Similarly, the keystone distortion correction signal generation portion 32 R can calculate the 
image formation magnification 0 of the right side imaging optical system 1 1 R on the basis of a signal from 
the focus encoder 42 R . Furthermore, the keystone distortion correction signal generation portions 32 R and 
32 L can calculate inclination angles 6 on the basis of signals from the corresponding rotational angle 

25 detecting means 28 R and 28 L . 

As shown in Fig. 5B. the multi-lens imaging apparatus of this embodiment further includes the keystone 
distortion correction signal generation portion 32 L for generating a keystone distortion correction signal on 
the basis of the output signals from the rotational angle detecting means 28 L , the zoom encoder 41 L , and 
the focus encoder 42 L of the left side imaging optica! system 11 L , and outputting the keystone distortion 

30 correction signal to the camera tube electron beam scanning locus control driver 31 L , the keystone 
distortion correction signal generation portion 32 R for generating a keystone distortion correction signal on 
the basis of the output signals from the rotational angle detecting means 28 R , the zoom encoder 41 R , and 
the focus encoder 42 R of the right side imaging optical system 11 R , and outputting the keystone distortion 
correction signal to a camera tube electron beam scanning locus control driver 31 R , and a synthesizing 

35 process portion 38 for forming a single high-fine image by synthesizing a right side image l R sent from a 
right side image sensor 12 R and a left side image l L sent from the left side image sensor 12 L . Note that the 
high-fine image formed by the synthesizing process portion 38 is output to a high fine monitor 40, and is 
displayed thereon. 

The principle of correcting keystone distortions in the keystone distortion correction signal generation 

40 portions 32 R and 32 L will be described below with reference to Figs. 6 to 8. 

When keystone distortions are respectively generated in the left side image l L obtained by the left side 
image sensor 12 L constituting the left side imaging system 10 L and in the right side image l R obtained by 
the right side image sensor 12 R constituting the right side imaging system 10 R , if a point A on the Y-axis of 
a required image I indicated by a broken line in Fig. 6 is defined as a reference point, a point A L on the left 

45 side image l L corresponding to the point A is deviated leftward in Fig. 6 by a horizontal deviation amount 
Ay, and a point A R on the left side image l R corresponding to the point A is deviated rightward in Fig. 6 by 
a horizontal deviation amount Ay. 

If an arbitrary point P 0 on the required image I indicated by a broken line in Fig. 7 is defined as a 
reference point, a point Pi on the left side image l R corresponding to the point P 0 is deviated rightward in 

50 Fig. 7 by a horizontal deviation amount Ayo' and is deviated upward in Fig. 7 by a vertical deviation amount 
Azo* if the coordinates of the point Po are represented by (yo', zo'), and the coordinates of the point Pi are 
represented by (yo' + Ayo\ zo' + Azo'). However, the horizontal and vertical deviation amounts Ayo' and 
Azo' can be calculated from the focal length f and the image formation magnification 0 of the right side 
imaging optical system 11 R and the inclination angle 0 of the right side imaging system 10 R . This principle 

55 will be described in more detail below with reference to Fig. 8. 

As shown in Fig. 8, when the right and left imaging systems 10 R and 10 L respectively perform imaging 
operations of an object surface 2, so that their optical axes L R and L L cross at a point O on the object 
surface 2, a conjugate object surface 3 passing through the point O on the object surface 2 and conjugate 
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with the right side image sensor 12 R is assumed. 

A point P 0 on the object surface 2 and a point Pi on the conjugate object surface 3, which satisfy OP 0 
= OPi , will be examined below. In this case, an object height y 0 of the point P 0 with respect to the right 
side imaging optical system 11 R and an object height yi of the point Pi with respect to the right side 
5 imaging optical system 11 R satisfy: 

yi = yo x cos(e (1) 

A length OQi between an intersection Qi of a perpendicular drawn from the point Pi to the optical axis L R| 
w and point 0 is expressed by: 

OQi = y 0 X sin(0) (2) 

Furthermore; if image heights on the right side image sensor 12 R with respect to the object heights y 0 and; 
15 yi are respectively represented by y 0 ' and yi\ the horizontal deviation amount Ay 0 ' is given by: 

Ay 0 ' = yr - yo* (3) 

If the distance (object distance) from the front-side principal point of the right side imaging optical system 
20 11 R to the object surface 2 is represented by So, and the distance from the rear-side principal point of the 
right side imaging optical system 11 R to the image surface of the right side image sensor 12 R is 
represented by So' the image height yi ' is expressed by: 

yi' = yi x So7(S 0 - OQi) (4) 

25 

Thus, from equations (1), (2), and (4), the image height yr can be expressed by: 
yi' = So* x y 0 x cos(0)/(So - yo x sin(0)) (5) 
30 Also, the image height yo' is expressed by: 
yo* = yo x SoVSo (6) 

For this reason, from equations (3), (5), and (6), the horizontal deviation amount Ay 0 ' can be expressed by: 

35 

Ayo' = {So 1 x cos(0)/(So' - yo 1 x sin(0)) - 1} x y 0 ' (7) 

Similarly, the vertical deviation amount Az 0 ' can be expressed by: 

40 

Az 0 ' = Zl ' - Zo « 

- {S 0 V(S 0 ' - y 0 ' x sin(6)) - 1} x z 0 ■ (8) 

45 

In this case, the distance So' from the rear-side principal point of the right side imaging optical system 
11 R to the image surface of the right side image sensor 12 R is expressed as follows using the focal length f 
and the image formation magnification 0 of the right side imaging optical system 1 1 R : 

50 So" = f x (1 - £) (9) 

Therefore, the horizontal and vertical deviation amounts Ay 0 ' and Azo' can be respectively calculated from 
the focal length f and the image formation magnification 0 of the right side imaging optical system 11 R and 
the inclination angle 6 of the right side imaging system 10 R . The same applies to the left side imaging 
55 system 10 L . 

The operation of the multi-lens imaging apparatus of this embodiment will be described below. 
When the object surface 2 is located at an infinity distance position, no keystone distortions are 
generated in the right and left side images l R and l L obtained by the right and left side image sensors 12 R 

8 



BNSDCCID: <EP. 



05S3737A1 J_> 



EP 0 563 737 A1 

and 12 L . For this reason, the synthesizing process portion 38 synthesizes the right and left side images l R 
and l L without executing the keystone distortion correction of the right and left side images l R and l L , and 
outputs the synthesized image to the high fine monitor 40. 

On the other hand, when the object surface 2 is located at a finite distance position, the right and left 

5 side imaging systems 10 R and 10 L are inclined in opposite directions by the convergence angle motors 27 R 
and 27 L , so that the right and left side imaging systems 10 R and 10 L are directed toward the object surface 
2. Thereafter, the inclination angles 6 of the right and left side imaging systems 10 R and 10 L are controlled 
by the convergence angle motors 27 R and 27 L , so that the sampling points on the optical axis L L of the left 
side imaging system 10 L are shifted by a 1/2 spatial phase pitch from those on the optical axis L R of the 

w right side imaging system 10 R . At this time, in synchronism with control of the inclination angles 0 by the 
convergence angle motors 27 R and 27 L , the magnification setting operations and focusing operations of the 
right and left side imaging optical systems 11 R and 11 L are performed using the zoom motors 25 R and 25 L 
and the focus motors 26 R and 26 L . 

In this case, since the imaging operations are performed in a state wherein the right and left imaging 

75 systems 10 R and 10 L are inclined through the inclination angle 9, keystone distortions are generated in the 
right and left side images i R and l L obtained by the right and left side image sensors 12 R and 12 L . 
Therefore, in order to correct these keystone distortions, the keystone distortion correction signal generation 
portion 32 L of the left side imaging optical system 11 L calculates the horizontal and vertical deviation 
amounts Ay 0 * and Az 0 * of each point on the left side image l L obtained by the left side image sensor 12 L 

20 using the output signals from the rotational angle detecting means 28 L and the focus encoder 42 L and the 
zoom encoder 41 L from equations (7) to (9) described above, generates a keystone distortion correction 
signal for correcting the keystone distortion generated in the left side image l L . and outputs the signal to the 
camera tube electron beam scanning locus control driver 31 L of the left side imaging optical system 11 L . In 
this case, the distance So' is calculated from equation (9). Alternatively, the distance So* may be calculated 

25 from the output signal from the focus encoder 42 L . The keystone distortion correction signal generation 
portion 32 R of the right side imaging optical system 1 1 R operates in the same manner as described above. 

The camera tube electron beam scanning locus control driver 31 L of the left side imaging optical 
system 11 L changes the scanning locus of the electron beam emitted from the camera tube constituting the 
left side image sensor 12 L on the basis of the keystone distortion correction signal sent from the keystone 

30 distortion correction signal generation portion 32 L , thereby correcting the keystone distortion generated in 
the left side image l L - More specifically, the camera tube electron beam scanning locus control driver 31 L of 
the left side imaging optical system 11 L changes the scanning locus of the electron beam by controlling a 
deflection circuit of the camera tube constituting the left side image sensor 12 L on the basis of the keystone 
distortion correction signal sent from the keystone distortion correction signal generation portion 32 L , 

35 thereby scanning the left side image l L along a point Ai — a point Bi , a point A2 — a point B 2 , a point A3 
—a point B 3 , and a point A4 — a point B*. as indicated by the left-hand side figure in Fig. 9. As a result, the 
left side image l L output from the camera tube constituting the left side image sensor 12 L is converted into 
the required image I, which is scanned along a point ai — a point bi , a point a 2 — a point b2, a point a 3 — 
a point b3, and a point a* — > a point b*. as indicated by the right-hand side figure in Fig. 9, and the 

40 converted image is input to the synthesizing process portion 38. 

In place of the rotational angle detecting means 28 R and 28 L , means for detecting relative postures of 
the right and left imaging systems 10 R and 10 L to obtain the inclination angles $ may be used. The right 
and left imaging optical systems 11 R and 11 L may be constituted using zoom lenses or fixed focal point 
lenses. 

45 Fig. 10 is a perspective view showing the arrangement of the right and left side imaging systems 10 R 
and 10 L in a multi-lens imaging apparatus according to another embodiment of the present invention, and 
Fig. 11 is a block diagram tor explaining operations of correction conversion portions 52 R and 52 L . 

The multi-lens imaging apparatus of this embodiment has the following differences from the multi-lens 
imaging apparatus of the embodiment shown in Figs. 5A and 5B. 
50 (1) Both the right and left side image sensors 12 R and 12 L comprise solid-state imaging device in place 
of the camera tubes. 

(2) The apparatus of this embodiment does not have the camera tube electron scanning locus control 
drivers 31 R and 31 L of the right and left imaging systems 10 R and 10 L shown in Fig. 5A. 

(3) The apparatus of this embodiment has a left side image memory 51 L for left side image data output 
55 from the left side image sensor 12 L , and a right side image memory 51 R for right side image data output 

from the right side image sensor 12 R , as shown in Fig. 11. 

(4) The apparatus of this embodiment has a correction conversion portion 52 L (to be described later) 
between the left side image memory 51 L and the synthesizing processing portion 38, and also has a 
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correction conversion portion 52 R (to be described later) between the right side image memory 51 R and 
the synthesizing processing portion 38. 
The operation of the multi-lens imaging apparatus of this embodiment will be described below. 
When the object surface 2 is located at an infinity distance position, since no keystone distortions are 

5 generated in the right and left side images l R and l L obtained by the right and left side image sensors 12 R 
and 12 L , right and left side image data respectively stored in the right and left image memories 51 p and 51 L 
are directly supplied to the synthesizing process portion 38 without executing keystone distortion correction 
of the right and left side images l R and l Ll and the right and left side images l R and l L are synthesized by the 
synthesizing process portion 38. The synthesized image is then output to the high fine monitor 40. 

io On the other hand, when the object surface 2 is located at a finite distance position, the right and left 

side imaging systems 10 R and 10 L are inclined by the convergence angle motors 27 R and 27 L in opposite 
directions, so that the right and left side imaging systems 10 R and 10 L are directed toward the object 
surface 2. Thereafter, the inclination angles e of the right and left side imaging systems 10 R and 10 L are 
controlled by the convergence angle motors 27 R and 27 L , so that the sampling points on the optical axis L L 

75 of the left side imaging system 10 L are shifted by a 1/2 spatial phase pitch from those on the optical axis L R 
ofthe right side imaging system 10 R . At this time, in synchronism with control of the inclination angles 0 by 
the convergence angle motors 27 R and 27 L , the magnification setting operations and focusing operations of 
the right and left side imaging optical systems 11 R and 11 L are performed using the zoom motors 25 R and 
25 L and the focus motors 26 R and 26 L . 

20 In this case, since the imaging operations are performed in a state wherein the right and left imaging 

systems 10 R and 10 L are inclined through the inclination angle 6, keystone distortions are generated in the 
right and left side images l R and l L obtained by the right and left side image sensors 12 R and 12 L . 
Therefore, in order to correct these keystone distortions, as shown in Fig. 11, the keystone distortion 
correction signal generation portion 32 L of the left side imaging optical system 11 L calculates the horizontal 

25 and vertical deviation amounts Ayo' and Azo' of each point on the left side image l L obtained by the left 
side image sensor 12 L using the output signals from the rotational angle detecting means 28 L and the focus 
encoder 42 L and the zoom encoder 41 L from equations (7) to (9) described above, generates a keystone 
distortion correction signal for correcting the keystone distortion generated in the left side image l L , and 
outputs the signal to the correction conversion portion 52 L . The correction conversion portion 52 L converts 

30 the coordinate of the left side image data of the left side image l L read out from the left side image memory 
51 L so as to cancel the horizontal and vertical deviation amounts Ay 0 ' and Az 0 ' on the basis of the keystone 
distortion correction signal sent from the keystone distortion correction signal generation portion 32 L , 
thereby correcting the keystone distortion generated in the left side image l L . In this case, the distance So' 
is calculated from equation (9). Alternatively, the distance So 1 may be calculated from the output signal from 

35 the focus encoder 42 L . The keystone distortion correction signal generation portion 32 R and the correction 
conversion portion 52 R of the right side imaging optical system 11 R perform the same operation as 
described above. 

As a result, the synthesizing process portion 38 synthesizes the right and left side image data of the 
right and left side images l R and l L , which are respectively sent from the correction conversion portions 52 R 

40 and 52 L , and in which the keystone distortions are corrected, so that a single high-fine synthesized image 
can be displayed on the high fine monitor 40. 

In the multi-lens imaging apparatus of this embodiment, the output signal from the correction conversion 
portion 52 L is supplied to the synthesizing process portion 38. Alternatively, the output signal from the 
correction conversion portion 52 L may be stored again in the left side image memory 51 l, and thereafter, 

45 the data may be supplied from the left side image memory 51 L to the synthesizing process portion 38. The 
same applies to the output signal from the other correction conversion portion 52 R . 

Fig. 12 is a perspective view showing an arrangement of the right _and left imaging systems 10 R and 10 L 
in a multi-lens imaging apparatus according to still another embodiment of the present invention, and Fig. 
13 is a block diagram for explaining operations of correction optical systems 61 R and 61 L . 

50 In the multi-lens imaging apparatus of this embodiment, lenses constituting correction optical systems 
61 R and 61 L (to be described later) are decentered in the vertical direction with respect to the optical axes 
L R and Lj_ so as to generate distortions, thereby correcting keystone distortions generated in the right and 
left side images l R and l L obtained by the right and left side image sensors 12 R and 12 L . More specifically, 
the arrangement of the multi-lens imaging apparatus is substantially the same as that of the multi-lens 

55 imaging apparatus of the embodiment shown in Figs. 10 and 11, except that the apparatus of this 
embodiment has the correction optical system 61 L arranged between the left side imaging optical system 
11 L and the left side image sensor 12 L , and the correction optical system 61 R arranged between the right 
side imaging optical system 11 R and the right side image sensor 12 R , in place of the correction conversion 
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portions 52 L and 52 R . 

The operation of the multi-lens imaging apparatus of this embodiment will be described below. 
When the object surface 2 is located at. an infinity distance position, since no keystone distortions are 
generated in Ihe right and left side images l R and l|_ obtained by the right and left side image sensors 12 R 

5 and 12 L , the right and left side images l R and l L are synthesized by the synthesizing process portion 38 
without executing keystone distortion correction of the right and left side images Ir and i L> and the 
synthesized image is output to the high fine monitor 40. 

On the other hand, when the object surface 2 is located at a finite distance position, the right and left 
side imaging systems 10 R and 10 L are inclined by the convergence angle motors 27 R and 27 L in opposite 

io directions, so that the right and left side imaging systems 10 R and 10 L are directed toward the object 
surface 2. Thereafter, the inclination angles 0 of the right and left side imaging systems 10 R and 10 L are 
controlled by the convergence angle motors 27 R and 27 L , so that the sampling points on the optical axis L L 
of the left side imaging system 10 L are shifted by a 1/2 spatial phase pitch from those on the optical axis L R 
ofthe right side imaging systam 10 R . At this time, in synchronism with control of the inclination angles 9 by 

15 the convergence anglo motors 27 R and 27 L , the magnification setting operations and focusing operations of 
the right and left side imaging optical systems 11 R and 11 L are performed using the zoom motors 25 R and 
25 L and the focus motors 26 R and 26 L . 

In this case, since the imaging operations are performed in a state wherein the right and left imaging 
systems 10 R and 10 L are inclined through the inclination angle 0, keystone distortions are generated in the 

20 right and left side images l n and l L obtained by the right and left side image sensors 12 R and 12 L . 
Therefore, in order to correct these keystone distortions, as shown in Fig. 13, the keystone distortion 
correction signal generation portion 32 L of the left side imaging optical system 11 L calculates the horizontal 
and vertical deviation amounts Ayo' and Az 0 ' of each point on the left side image l L obtained by the left 
side image sensor 12 L using the output signals from the rotational angle detecting means 28 L and the focus 

25 encoder 42. and the zoom encoder 41 L from equations (7) to (9) described above, generates a keystone 
distortion correction signal for correcting the keystone distortion generated in the left side image l L , and 
outputs the signal to the correction optical system 61 L of the left side imaging optical system 11 L . The 
correction optical system 61, of the left side imaging optical system 11 L decenters at least one of lenses 
constituting the correction optical system 61 L in the vertical direction with respect to the optical axis L t on 

30 the basis of the keystone distortion correction signal sent from the keystone distortion correction signal 
generation portion 32 L so as to generate a distortion, thereby correcting the keystone distortion generated in 
the left side image 1 L - In this case, the distance So* is calculated from equation (9). Alternatively, the 
distance So' may be calculated from the output signal from the focus encoder 42 L . The keystone distortion 
correction signal generation portion 32 R and the correction optical system 61 R of the right side imaging 

35 optical system 11 R perform the same operations as described above. 

The principle of correcting keystone distortions will be described below with reference to Fig. 14. 
The object surface 2 is inclined through the inclination angle e with respect to the optical axis L R , and 
its image is formed on the right side image sensor 12 R . If the distance from the object surface 2 to the right 
side imaging optical system 11 R is represented by Si, the distance from the right side imaging optical 

40 system 11 R to the image surface of the right side image sensor 12 R is represented by Si and a point O on 
the object surface 2 is defined as an intersection between the optical axis L R and the object surface 2, an 
image formation magnification fi H at a height H of the object surface 2 is expressed as follows by a paraxial 
calculation: 

45 

Ph ■ S 1 , /(S X + H x sin(6)) 

- P * {S 1 /(S 1 + H x sin(9))} (10) 

-50 

where 

0 = the image formation magnification at the point O on the object surface 2 

A decentering distortion d generated when a concave lens 62 R of the correction optical system 61 R - 
(consisting of the concave lens 62 R and a convex lens 63 R ) arranged between the right side imaging- optical 
55 system 11r and the right side image sensor 12 R is parallelly decentered by a distance E in the vertical 
direction in Fig. 15 is given, in consideration of up to a third-order term, by: 

d = -(E/2) x (tan(a))) 2 x (3 x V E1 - V E2 ) (11) 
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for tan(a>) = H/f (f = the focal length of the right side imaging optical system 1 1R) 
Note that V E1 and V E2 included in equation (11) are parallel decentering distortion coefficients of the 
correction optical system 61 R , and are respectively given by: 

5 

V E i = (h x <p x V q - a x V p ) - (h x * x B q - a x B p ) (12) 
V E2 = h x <f> x P p - a x P p (13) 
where 

4> = the refracting power of the correction optical system 61 r 
V p = the third-order distortion of the convex lens 63 R 
Bp = the astigmatism coefficient of the convex lens 63 R 
P p = the Petzval's sum of the convex lens 63 R 

V q = the third-order distortion on the object surface 2 side of the concave lens 62 R 
B q = the astigmatism coefficient on the object surface 2 side of the concave lens 62 R 
P q = the Petzval's sum on the object surface 2 side of the concave lens 62 R 
h = the incident position of the paraxial object ray onto the correction optical system 61 R 
a = the incident angle of the paraxial object ray onto the correction optical system 61 R 
Since the distortion generated when the correction optical system 61 R is decentered is calculated in the 
above-mentioned procedures, keystone distortion correction shown in Fig. 16 can be attained by decen- 
tering the correction optical system 61 R so that the signs of & H /& - 1 and the decentering distortion d are 
opposite to each other: 

25 d = - 1 (14) 

In this manner, a keystone distortion can be corrected in a region up to the third order. Note that the parallel 
decentering amount E of the correction optical system 61 R can be calculated from equations (10) to (14). 
When the right side image l R in which the keystone distortion is corrected in this manner, and which is 
30 obtained from the right side image sensor 12 R , and the left side image l L in which the keystone distortion is 
similarly corrected, and which is obtained from the left side image sensor 12 L are synthesized by the 
synthesizing process portion 38, a single high-fine image can be displayed on the high fine monitor 40 (see 
Fig. 13). 

Figs. 17A and 17B show a multi-lens imaging apparatus according to still another embodiment of the 
35 present invention. 

The multi-lens imaging apparatus of this embodiment obtains a single high-fine image by synthesizing 
two images obtained by picking up an image of a common object using two imaging systems (right and left 
side imaging systems 10 R and 10 L ). 

The arrangement of the right and left side imaging systems 10 R and 10l will be described in detail 

40 below with reference to Fig. 17A. 

The left side imaging system 10 L includes lens groups 21 L to 24 L including a zoom lens group 22 u and 
a focus lens group 24 L , a zoom motor 25 L as a driving system for driving the zoom lens group 22 L , a focus 
motor 26 L as a driving system for driving the focus lens group 24 L , a mechanism system (not shown) and a 
driving system (convergence angle motor 27 L ) for integrally rotating the left side imaging optical system 11 L 

45 and a left side image sensor 12 L in a plane including an optical axis L L , and a rotational angle detecting 
means 28 L for detecting the rotational angle of the convergence angle motor 27 L , which constitute a left side 
imaging optical system 11 L . As the rotational angle detecting means 2B L , for example, an externally attached 
member such as a rotary encoder may be used, or a means such as a pulse motor for detecting the 
rotational angle by the driving system itself according to a driving method may be used. The right side 

so imaging system 10 R has the same arrangement as that of the left side imaging system 10 L . 

Assume that when the focus motor 26 L and the zoom motor 25 L of the left side imaging optical system 
11 L are independently controlled by signals from the focus encoder 42 L and the zoom encoder 41 L , and a 
focus motor 26 R and a zoom motor 25 R of a right side imaging optical system 11 R are independently 
controlled by signals from a focus encoder 42 R and a zoom encoder 41 R , a focal length f of the left side 

55 imaging optical system 11 L always coincides with a focal length f of the right side imaging optical system 
11 R , and an image formation magnification & of the left side imaging optical system 11 L always coincides 
with an image formation magnification 0 of the right side imaging optical system 11 R . 
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At this time, a calculation control portion 71 shown in Fig. 17B can calculate the focal length f of the left 
side imaging optical system 1 1 L on the basis of a signal from the zoom encoder 41 L , and can calculate the 
focal length f of the right side imaging optical system 1 1 R on the basis of a signal from the zoom encoder 
41 R . Also, since the calculation control portion 71 can calculate an object distance to the left side imaging 

5 optical system 11 L on the basis of a signal from the focus encoder 42 L , it can calculate the image formation 
magnification 0 of the left side imaging optical system 11 L> and similarly, it can calculate the image 
formation magnification 0 of the right side imaging optical system 11r on the basis of a signal from the 
focus encoder 42 R . Furthermore, the calculation control portion 71 can calculate inclination angles d from 
the signals output from the rotational angle detecting means 28 R and 28 L . 

to The output signals from the zoom encoders 41 R and 41 L and the focus encoders 42 R and 42 L are input 
to the calculation control portion 71, as shown in Fig. 17B. The calculation control portion 71 can calculate 
the distance So' from the rear-side principal point of the left side imaging optical system 11 L to the image 
surface of the left side image sensor 12 L and the distance So' from the rear-side principal point of the right 
side imaging optical system 11 R to the image surface of the right side image sensor 12 R from equation (9) 

75 described above: 

So 1 = f x (1 - 0) (9) 

Left side image data of a left side image l L output from the left side image sensor 12 L is stored in a left 

20 side image memory 72 L , and right side image data of a right side image l R output from the right side image 
sensor 12 R is stored in a right side image memory 72 R . 

The principle of correcting keystone distortions generated in the right and left images l R and l L by a 
keystone distortion conversion portion 73. second right and left side image memories 74 R and 74 L , and a 
subtraction process circuit 75 shown in Fig. 17B will be described below with reference to Figs. 18 and 19. 

25 The keystone distortion conversion portion 73 calculates horizontal and vertical deviation amounts Ay 0 ' 
and Azo f of predetermined several image points from equations (7) and (8) described above using the left 
side image data read out from the left side image memory 72 Lt the right side image data read out from the 
right side image memory 72 R , and the distances So' sent from the calculation control portion 71. At this 
time, the inclination angle 6 is defined as a conversion variable (an angle is not detected), and the horizontal 

30 and vertical deviation amounts Ayo' and Azo' are calculated by substituting, e.g., B - By (By is an arbitrary 
value). Thereafter, the keystone distortion conversion portion 73 shifts the coordinates of the image points in 
the right and left side image memories 72 R and 72 L by the calculated horizontal and vertical deviation 
amounts Ay 0 ? and Azo', and writes the shifted right and left side image data in the second right and left 
side image memories 74 R and 74 L , respectively. Thereafter, the subtraction process circuit 75 reads out the 

35 coordinate-shifted right and left side image data respectively from the second right and left side image 
memories 74 R and 74 L , and calculates a sum (to be referred to as an "image difference" hereinafter) 5 S1 of 
the absolute values of differences of these data. Subsequently, the keystone distortion conversion portion 
73 changes the inclination angle 6 (conversion variable) to be 6 = 02, and an image difference 5^ is 
similarly calculated. 

40 When the above-mentioned operations are repeated, since the relationship between the inclination angle 
0 (conversion variable) and the image difference 6 S can be obtained, as shown in, e.g., Fig. 19, an inclination 
angle Bo (conversion variable) at which the image difference 5 S assumes a minimum value is obtained. The 
keystone distortion conversion portion 73 calculates the horizontal and vertical deviation amounts Ay 0 ' and 
Azo' of all image points of the left side image data from equations (7) and (8) using the inclination angle Bo - 

45 (conversion variable) at which the image difference 5 S assumes a minimum value, thereafter, shifts the 
coordinates of all the image points of the left side image data by the calculated horizontal and vertical 
deviation amounts Ayo' and Azo', and writes the shifted left side image data in the second left side image 
memory 74 L . As a result, the left side image data in which the keystone distortion is corrected is written in 
the second left side image memory 74 L . 

so Therefore, when the left side image data in which the keystone distortion is corrected, and which is 
written in the second left side image memory 74 L , and the right side image data in which the keystone 
distortion is corrected, and which is written in the second right side image memory 74 R are synthesized by 
a synthesizing process portion (not shown), a single high-fine image can be displayed on a high fine 
monitor (not shown). 

55 Figs. 20A and 20B show a multi-lens imaging apparatus according to still another embodiment of the 

present invention. 

The arrangement of the multi-lens imaging apparatus of this embodiment is substantially the same as 
that of the multi-lens imaging apparatus of the embodiment shown in Figs. 17A and 17B, except that the 
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right and left image sensors 12 R and 12 L comprise camera tubes, and the keystone distortion conversion 
portion 73 controls camera tube electron beam scanning locus drivers 31 R and 31 L according to the 
calculated horizontal and vertical deviation amounts Ayo' and Azo'. 

Since the operations of the camera tube electron beam scanning locus drivers 31 r and 31 L and the 
5 principle of correcting keystone distortions in the multi-lens imaging apparatus of this embodiment are the 
same as those in the multi-lens imaging apparatuses in the embodiments shown in Figs. 5A and 5B and 
Figs. 17A and 17B, a repetitive description will be avoided here. 

Figs. 21 A to 21 C show a multi-lens imaging apparatus according to still another embodiment of the 
present invention, and Fig. 22 shows a basic arrangement of imaging systems in the multi-lens imaging 
10 apparatus of the embodiment shown in Figs. 21 A to 21 C. 

The multi-lens imaging apparatus of this embodiment obtains a single high-fine image by synthesizing 
two images obtained by picking up an image of a common object using two sets of imaging systems (right 
and left side imaging systems 10 R and 10 L ) . 

As shown in Fig. 22. the left side imaging system 10 L is constituted by a left side imaging optical 
15 system 11 L and a left side image sensor 12 L , and the right side imaging system 10 R is constituted by a right 
side imaging optical system 1 1 R and a right side image sensor 12 R . The right and left side imaging optical 
systems 11 R and 11 L have equivalent specifications, and each comprises a zoom lens. The right and left 
side image sensors 12 R and 12 L have equivalent specifications, and each comprises a camera tube such as 
a saticon or a solid-state imaging element such as a CCD. Optical axes L R and L L of the right and left 
20 imaging systems 10 R and 10 L cross almost at a point 0 on an object surface 2, and these systems are 
arranged at positions axially symmetrical about a normal O-O' to the object surface 2. 

In this multi-lens imaging apparatus, when the object distance is changed, the convergence angle 26 is 
changed by rotating each of the right and left side imaging systems 10 R and 10 L about an X mark shown in 
Fig. 22 in accordance with the change in object distance, thus performing an imaging operation. 
25 The arrangement of the right and left imaging systems 10 R and 10 L will be described in detail below 
with reference to Fig. 21 A. 

The left side imaging system 10 L includes lens groups 21 L to 24 L including a zoom lens group 22 L and 
a focus lens group 24 L , a zoom motor 25 L as a driving system for driving the zoom lens group 22 L , a focus 
motor 26 L as a driving system for driving the focus lens group 24 L , a mechanism system (not shown) and a 
30 driving system (convergence angle motor 27 L ) for integrally rotating the left side imaging optical system 11 L 
and a left side image sensor 12 L in a plane including an optical axis L L , and a rotational angle detecting 
means 28 L for detecting the rotational angle of the convergence angle motor 27 L , which constitute a left side 
imaging optical system 1 1 L . As the rotational angle detecting means 28 L , for example, an externally attached 
member such as a rotary encoder may be used, or a means such as a pulse motor for detecting the 
35 rotational angle by the driving system itself according to a driving method may be used. The right side 
imaging system 10 R has the same arrangement as that of the left side imaging system 10 L . 

A control method for the convergence angle motors 27 R and 27 L will be described below with reference 
to Fig. 21 B. 

When the right and left imaging systems 10 R and 10 L perform imaging operations respectively at 
40 convergence angles 2e+A0 R and 20+A0 L to have angular deviations A0 R and A0 L with respect to the 
predetermined convergence angle 20 (see Fig. 22), the rotational angle detecting means 28 L detects the 
angular deviation A9 L with respect to an angle 0 (see Fig. 22) formed between the optical axis L L of the left 
side imaging system 10 L and the normal O-O* to the object surface 2, and the rotational angle detecting 
means 28 R detects the angular deviation A0 R with respect to an angle $ formed between the optical axis R R 
45 of the right side imaging system 10 R and the normal 0-0' to the object surface 2. The angular deviations 
A6 R and A0 L output from the rotational angle detecting means 28 R and 28 L are supplied to a control portion 
231 . 

Assume that the convergence angle motors 27 R and 27 L have the following response characteristics. 
For the sake of simplicity, as shown in Fig. 23, only a mis-registration amount Aa of a image output 
so from the right side imaging system 10 R when the optical axis L R of the right side imaging system 10 R is 
deviated by the angular deviation A0 R will be examined below. If the distance (object distance) from the 
front-side principal point of the right side imaging optical system 11 R to the object surface 2 is represented 
by So, the distance from the rear-side principal point of the right side imaging optical system 1 1 R to the 
right side image sensor 12 R is represented by Si, and the deviation amount of the optical axis L R of the 
55 right side imaging system 10 R in the object surface 2 is represented by Ay, when the angular deviation A0 R 
is small, we have: 

S c x tan(A0 R ) = Ay x cos(0) (15) 
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If the image formation magnification of the right side imaging optical system 11 R is represented by £, we 
have: 

5 0 = Aa'Ay (16) 

Furthermore, if the focal length of the right side imaging optical system 11 R is represented by f, we have: 
$u = f - (0 • 1K0 (17) 

From equations (15) to (17), the mis-registration amount Aa of the image is given by: 
A3 = {f x (0 > 1) x tan(A0 R )}/cos(0) (18) 

75 Therefore, assume that the convergence angle motor 27 R shown in Fig. 21 A has sufficient response 
characteristics for an angle (A0 R ) necessary for correcting the mis-registration amount Aa of the image 
given by equation (18) to be equal to or smaller than an allowable value Aao. The same applies to the 
convergence angle motor 27 L of the left side imaging optical system 11 L . 

The control portion 231 supplies a correction signal -A0 L to the convergence angle motor 27 L of the left 

20 side imaging system 10 L as a control target value, and supplies a correction signal -A0 R to the convergence 
angle motor 27 n of the right side imaging system 10 R as a control target value, as shown in Fig. 21 B. As a 
result, the mis-registration amount Aa of the image obtained by each of the right and left imaging systems 
10 R and 10 L can be corrected to be equal to or smaller than the allowable value Aao. 

When the mis-registration amounts of the images are corrected as described above, a image signal 

25 output from the left side image sensor 12 L is stored in a left side image memory 232 L , and a image signal 
output from the right side image sensor 12 R is stored in a right side image memory 232 R , as shown in Fig. 
21 C. Thereafter, these images are synthesized by a synthesizing process circuit 233, and a synthesized 
image signal is output. 

Therefore, since the multi-lens imaging apparatus of this embodiment can synthesize images obtained 
30 by the right and left imaging systems 10 R and 10 L while registering the two images, if, for example, the 
convergence angle is controlled in advance so that the relative spatial phase between the right and left side 
image sensors 12 R and 12 L is shifted by a 1/2 pitch when the sampling points of the right and left image 
side sensors 12 R and 12 L are imaginably projected onto an object, the synthesizing process circuit 233 can 
obtain a single high fine image. 
35 In the multi-lens imaging apparatus of this embodiment, when a solid state imaging element is used as 
each of the right and left side image sensors 12 R and12 L , either a single-sensor type element or two- or 
three-sensor type element via a color separation system may be used. When a camera tube is used as 
each of the right and left side image sensors 12 R and 12 Ll either a single-tube type element or two- or 
three-tube type element via a color separation system may be used. The control portion 231 may supply a 
40 correction signal -(A0 L + A0 R ) to one of the two convergence angle motors 27 R and 27 L as a control target 
value in place of respectively supplying the correction signals -Aa R and -A0 L to the two convergence angle 
motors 27 R and 27 L ascontrol target values. 

Figs. 24A and 24B show a multi-lens imaging apparatus according to still another embodiment of the 
present invention. 

45 In the multi-lens imaging apparatus of the embodiment shown in Figs. 21 A to 21 C, as can be 
understood from equation (18), when the focal lengths f of the right and left side imaging optical systems 
11 R and 11 L are large, or when the allowable values Aao of the mis-registration amounts of images are 
small, the correction signals -A0 R and -A0 L output from the control portion 231 also take small values (e.g., 
on the order of several seconds), and it becomes difficult to obtain angular response performance 

so necessary for correction under the influence of, e.g., non-sensitive bands of the convergence angle motors 
27 R and 27 L . 

In the multi-lens imaging apparatus of the embodiment shown in Figs. 24A and 24B, the image mis- 
registration amount Aa given by equation (18) is calculated, and is corrected by means other than the 
convergence angle motors 27 R and 27 L . thereby solving the above-mentioned problem. The apparatus of 
55 this embodiment has the following differences from the multi-lens imaging apparatus of the embodiment 
shown in Figs. 21 A to 21 C. 

(1) Zoom encoders 41 R and 41 L for detecting the focal lengths f of the right and left side imaging optical 
systems 1 1 R and 1 1 L are respectively provided to the zoom motors 25 R and 25 L . 
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(2) Focus encoders 42 R and 42 L for detecting the object distances So with respect to the right and left 
side imaging optical systems 1 1 R and 11 L are respectively provided to the focus motors 26 R and 26 L . 

(3) The apparatus of this embodiment includes horizontal direction drive systems 243 R and 243 L for 
respectively driving the right and left side image sensors 12 R and 12 L in the horizontal direction. 

5 (4) The apparatus of this embodiment also includes a calculation control portion 244 (to be described 
later; see Fig. 24B). 

The operation of the multi-lens imaging apparatus of this embodiment will be described below. 
In the left side imaging optical system 11 L , when the zoom motor 25 L is controlled by a signal from the 
zoom encoder 41 L , and the focus motor 26 L is controlled by an output signal from the focus encoder 42 L> 

io the focal length f and the image formation magnification £ are caused to always coincide with each other. In 
the right side imaging optical system 11 Rt the focal length f and the image formation magnification 0 are 
similarly caused to always coincide with each other. 

Therefore, the focal lengths f and the image formation magnifications £ of the right and left side 
imaging optical systems 11 R and 11 L can be obtained from the output signals from the zoom encoders 41 R 

75 and 41 Lf and the object distances So with respect to the right and left side imaging optical systems 11 R and 
11 L can be obtained from the output signals from the focus encoders 42 R and 42 L . For this reason, the 
output signals from the zoom encoders 41 R and 41 L and the focus encoders 42 R and 42 L are input to the 
calculation control portion 244, and the calculation control portion 244 calculates equation (18) described 
above, thus obtaining the mis-registration amounts Aao of images. The calculation control portion 244 

20 controls the horizontal direction drive systems 243 R and 243 L to move the right and left side image sensors 
12 R and 12 L in the horizontal direction according to the calculated mis-registration amounts Aao of the 
images, thereby correcting the mis-registration amounts Aa of the images to be equal to or smaller than the 
allowable value Aao. 

As each of the zoom encoders 41 R and 41 L and the focus encoders 42 R and 42 L , for example, an 
25 externally attached member such as a potentiometer may be used, or a means such as a pulse motor for 
detecting the rotational angle by the driving system itself according to a driving method may be used. Also, 
each of the horizontal direction drive systems 243 R and 243 L may comprise, e.g., a piezo element or 
piezoelectric bimorph. 

When a swing motion in the vertical direction occurs due to insufficient precision of a mechanism 
30 system (not shown) for driving the convergence angle motor, encoders for respectively detecting inclination 
angles (elevation angles) A<h, in the vertical direction, of the right and left side imaging systems 10 R and 
10 L , and vertical direction drive systems for respectively moving the right and left side image sensors 12 R 
and 12 L in the vertical direction may be separately arranged. Thus, when the right and left side image 
sensors 12 R and 12 L are moved in the vertical direction according to the detected inclination angles A<jn in 
35 the vertical direction, mis-registration amounts Ab, in the vertical direction, of images obtained by the right 
and left imaging systems 10 R and 10 L can be similarly corrected to be equal to or smaller than an allowable 
value Abo- At this time, the mis-registration amount Ab, in the vertical direction, of each image can be 
calculated by the following equation like in equation (18): 

40 Ab = f x (0 - 1) x tan(A<J>i) (19) 

For the following reasons, the mis-registration amounts Ab, in the vertical direction, of images may be 
corrected upon combination of coarse adjustment attained by the convergence angle motors 27 R and 27 L in 
the multi-lens imaging apparatus of the embodiment shown in Figs. 21 A to 21 C, and fine adjustment 
45 attained by the horizontal direction drive systems 243 R and 243 L in the multi-lens imaging apparatus of this 
embodiment: 

(1) to prevent regions to be subjected to aberration correction of the right and left side imaging optical 
systems 11 R and 11 L from being increased very much; and 

(2) to prevent the loads on the horizontal direction drive systems 243 R and 243 L from being increased. 

50 Figs. 25A and 25B show a multi-lens imaging apparatus according to still another embodiment of the 
present invention. 

In the multi-lens imaging apparatus of this embodiment, the right and left side image sensors 12 R and 
12 L respectively comprise camera tubes, and the arrangement of the apparatus of this embodiment is 
substantially the same as that of the multi-lens imaging apparatus of the embodiment shown in Figs. 24A 
55 and 24B, except that camera tube electron beam scanning locus control drivers 251 R and 251 L are used in 
place of the horizontal direction drive systems 243 R and 243 L . 

More specifically, in the multi-lens imaging apparatus of this embodiment, since the camera tubes read 
video signals by scanning electron beams, the scanning loci of the electron beams by the camera tubes are 
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controlled by the camera tube electron beam scanning locus control drivers 251 R and 251 L in accordance 
with the mis-registration amounts Aa obtained by equation (18) above, thereby registering the images 
expressed by the video signals output from the right and left side image sensors 12 R and 12 L . 

Figs. 26A and 26B show a multi-lens imaging apparatus according to still another embodiment of the 
5 present invention. 

In the multi-lens imaging apparatus of this embodiment, the right and left side image sensors 12 R and 
12 L respectively comprise solid-state imaging device, and the mis-registration amounts Aa of images 
obtained by the right and left imaging systems 10 R and 10 L are corrected to be equal to or smaller than the 
allowable value Aao without using the horizontal direction drive systems 243 R and 243 L shown in Fig. 24B. 
70 The apparatus of this embodiment has the following differences from the multi-lens imaging apparatus of 
the embodiment shown in Figs. 24A and 24B. 

(1) The apparatus of this embodiment does not include the horizontal direction drive systems 243 R and 
243 L . 

(2) The apparatus of this embodiment includes a correction conversion portion 261 and a second left 
75 side image memory 262 between the left side image memory 232 L and the synthesizing process circuit 

233 (see Fig. 26B). 

The operation of the multi-lens imaging apparatus of this embodiment will be described below. 
The calculation control portion 244 receives the output signals from the rotational angle detecting 
means 28 R and 28 L , the zoom encoders 41 R and 41 L , and the focus encoders 42 R and 42 L , and calculates 

20 the mis-registration amounts Aa and Ab of images using equations (18) and (19). The correction conversion 
portion 261 converts the coordinates of image data output from the left side image memory 232 L in 
accordance with the mis-registration amount Aa calculated by the calculation control portion 244, and writes 
the converted image data in the second left side image memory 262. As a result, when the synthesizing 
process circuit 233 synthesizes the image data respectively output from the second left side image memory 

25 262 and the right side image memory 232 R , a high-fine synthesized image signal can be obtained. 

Note that the correction conversion portion 261 and a second left side image memory may be arranged 
between the right side image memory 232R and the synthesizing process circuit 233 in place of the 
correction conversion portion 261 and the second left side image memory 262. The left side image memory 
232 L and the second left side image memory 262 may be replaced with a common one. Furthermore, when 

30 the coordinates of image data output from the left side image memory 232 L are converted, an interpolation 
calculation process of image data may be performed as needed. 

In each of the multi-lens imaging apparatus of the embodiments shown in Figs. 21 A to 26B according to 
the present invention, the mis-registration amounts Aa of images are calculated based on the output signals 
from the rotational angle detecting means 28 R and 28 L , the zoom encoders 41 R and 41 L , and the focus 

35 encoders 42 R and 42 L . Alternatively, the mis-registration amounts Aa of images may be calculated from 
image signals output from the right and left imaging systems 10 R and 10L. Such embodiments of multi-lens 
imaging apparatuses of the present invention will be described hereinafter. 

Fig. 27 is a block diagram for explaining an operation of a correlation calculation process portion in a 
multi-lens imaging apparatus according to still another embodiment of the present invention. 

40 The multi-lens imaging apparatus of this embodiment has the following differences from the multi-lens 
imaging apparatus of the embodiment shown in Figs. 26A and 26B. 

(1) The apparatus of this embodiment includes a correlation calculation process portion 271 for 
performing a correlation calculation between image data output from the second left side image memory 
262 and image data output from the right side image memory 232 R to obtain a correlation value S s - When 

45 the correlation value 6 S is equal to or larger than a predetermined value, the portion 271 calculates a new 
shift amount X so that the correlation value 6 S becomes equal to or smaller than the predetermined value, 
and outputs the calculated shift amount X to the correction conversion portion 261 . 

(2) When the correction conversion portion 261 writes image data output from the left side image 
memory 232 L in the second left side image memory 262, it shifts the coordinates of the image data in 

50 the horizontal and vertical directions by the shift amount X sent from the correlation calculation process 
portion 271. 

The operation of the multi-lens imaging apparatus of this embodiment will be described below. 

The correction conversion portion 261 shifts the coordinates of image data output from the left side 
image memory 232 L in the horizontal and vertical directions by a predetermined shift amount X (e.g., by 
55 one pixel each in the horizontal and vertical directions), and writes the coordinate-shifted image data in the 
second left side image memory 262. The correlation calculation process portion 271 performs a correlation 
calculation between image data output from the second left side image memory 262 and image data output 
from the right side image memory 232 R to obtain a correlation value 6 S . When the correlation value 6 S is 
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equal to or larger than a predetermined value, the portion 271 calculates a new shift amount X so that the 
correlation value 6 S becomes equal to or smaller than the predetermined value, and outputs the calculated 
shift amount X to the correction conversion portion 261. The correction conversion portion 261 shifts the 
coordinates of image data output from the left side image memory 232 L in the horizontal and vertical 

5 directions by the new shift amount X, and writes the coordinate-shifted image data in the second left side 
image memory 262. Thereafter, the above-mentioned operations are repeated until the correlation value a s 
becomes equal to or smaller than the predetermined value. After the correlation value 5 S becomes equal to 
or smaller than the predetermined value, a synthesizing process circuit (not shown) synthesizes image data 
respectively output from the second left side image memory 262 and the right side image memory 232 R , 

io thus obtaining a high-fine synthesized image signal. 

In order to shorten the correlation calculation time in the correlation calculation process portion 271, for 
example, the correlation calculation may be performed using only image data at the central portion of a 
frame. As a correlation calculation method, a known matching method, gradient method, or the like may be 
used. 

75 Since a finally obtained shift amount Xo indicates the mis-registration amount Aa of a image, the 
correlation calculation process portion 271 may calculate a correction signal corresponding to the finally 
obtained shift amount Xo, and the correction control methods described in the embodiments shown in Figs. 
21 A to 25B, i.e., the following control methods may be executed: 
(1) convergence angle control 
20 (2) correction drive control using drive systems capable of shifting the image sensors in the horizontal, 
and vertical directions 

(3) correction control for changing the electron beam scanning loci 
The above-mentioned multi-lens imaging apparatus of the embodiment shown in Fig. 27 executes 
correction control of the mis- registration amounts Aa of images caused by convergence angle errors using 
25 the image signals output from the right and left side imaging systems 10 R and 10 L . However, the correction 
control of the mis-registration amounts Aa of images caused by a difference between the image formation 
magnifications of the right and left imaging optical systems 10 R and 10 L may be executed using the image 
signals output from the right and left imaging systems 10 R and 10 L . A multi-lens imaging apparatus 
according to such an embodiment of the present invention will be described below. 
30 Fig. 28 is a block diagram for explaining an operation of a magnification correlation calculation process 
portion in a multi-lens imaging apparatus according to still another embodiment of the present invention. 

The multi-lens imaging apparatus of this embodiment has substantially the same arrangement as that of 
the multi-lens imaging apparatus of the embodiment shown in Fig. 27, except that the apparatus of this 
embodiment includes a magnification correlation calculation process portion 281, a calculation process 
35 portion 282, and a third left side image memory 283. 

The operation of multi-lens imaging apparatus of this embodiment will be described below. 
Assume that the central positions of images obtained by the right and left side imaging systems 10 R 
and 10 L are registered with each other like in the multi-lens imaging apparatus of the embodiment shown in 
Fig. 27. 

40 The calculation process portion 282 converts the coordinates of image data output from the left side 
image memory 232 L by multiplying the coordinates with a constant k proportional to the distance from the 
center of the frame, and writes the coordinate-converted image data in the third left side image memory 
283. The magnification correlation calculation process portion 281 executes a correlation calculation 
between image data output from the third left side image memory 283 and image data output from the right 

45 side image memory 232 R to obtain a correlation value 5 S . When the correlation value 6 S is equal to or larger 
than a predetermined value, the portion 281 calculates a new constant k, so that the correlation value 5 S 
becomes equal to or smaller than the predetermined value, and outputs the constant to the calculation 
process portion 282. The calculation process portion 282 converts the coordinates of image data output 
from the left side image memory 232 L by multiplying the coordinates with the new constant k, and writes 

so the coordinate-converted image data in the third left side image memory 283. Thereafter, the above- 
mentioned operations are repeated until the correlation value 6 S becomes equal to or smaller than the 
predetermined value. After the correlation value 6 S becomes equal to or smaller than the predetermined 
value, a synthesizing process circuit (not shown) synthesizes image data respectively output from the 
second left side image memory 262 and the right side image memory 232 thus obtaining a high-fine 

55 synthesized image signal from which the mis-registration amounts of images caused by a difference 
between the image formation magnifications of the right and left imaging optical systems 11 R and 11 L are 
corrected. 
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Since a finally obtained constant ko indicates the ratio of image formation magnifications of the right and 
left imaging optical systems 11 R and 11 L , the magnification correlation calculation process portion 281 may 
calculate a correction signal according to the finally obtained constant ko, and may execute the correction 
control methods described in the embodiments shown in Figs. 21 A to 25B. 

5 In order to shorten the correlation calculation time of the magnification correlation calculation process 

portion 281, the correlation calculation may be executed using only image signals in a hatched region (a 
region obtained by narrowing a horizontal process portion) having as its center a line Y-Y' passing through a 
point O on the object surface 2 and perpendicular to the plane including two optical axes L R and L L , as 
shown in Rg. 29. Since this region has symmetricity from the arrangement conditions of the right and left 

io side imaging optical systems 11 R and 11 L , and suffers from a common image distortion, a difference 
between the image formation magnifications can be easily calculated. 

In order to shorten the correlation calculation time of the magnification correlation calculation process 
portion 281, a positional deviation correlation in the vertical direction may be calculated by the method 
described in the embodiment shown in Figs. 26A and 26B using only image signals in two upper and lower 

75 hatched regions which have as their center a line Y-Y' passing through the point O on the object surface 2 
and perpendicular to the plane including two optical axes L R and L L , and which are separated from the point 
O, as shown in Fig. 30. When representative points of these regions are represented by Q and R, and when 
images of these representative points Q and Ft on the left side image sensor 12 L are represented by Q L * 
and R L ' and images of these representative points Q and R on the right side image sensor 12 R are 

20 represented by Q R ' and R R \ a ratio of a line Ql* - FV to a line Q R and R R is calculated to obtain the final 
constant ko. This technique utilizes the fact that the detection sensitivity of the difference between the 
image formation magnifications can be improved toward the periphery of the frame, and the fact that the 
difference between the image formation magnifications can be regarded as a positional deviation in a small 
region. 

25 The final constant ko is obtained as described above, and the following correction control can be made 
on the basis of this constant ko. 

(1) When the zoom motors 25 R and 25 L have response characteristics necessary for correcting the mis- 
registration amounts of images caused by the difference between the image formation magnifications to 
be equal to or smaller than a predetermined value (allowable value), servo control is executed by 

30 supplying a correction signal to one or both of the zoom motors 25 R and 25 L . 

(2) When it is difficult for the zoom motors 25 R and 25 L to obtain response characteristics necessary for 
correcting the mis-registration amounts due to the influence of the non-sensitive bands or the like, 
correction control by changing the scanning loci of the electron beams or correction control by 
converting the coordinates of the image memory is suitably executed. In this case, an interpolation 

35 calculation process of image data may be performed as needed. 

Each of the multi-lens imaging apparatuses according to the above-mentioned embodiments of the 
present invention comprises two imaging systems. However, the present invention can also be applied to an 
apparatus using three or more imaging systems. 

An embodiment of an apparatus for displaying images while partially overlapping image fields using a 
40 plurality of imaging systems, e.g., for obtaining a panoramic image or a 3D-2D mixed image is disclosed. 

Fig. 31 shows the basic arrangement of a panoramic multi-lens imaging system according to the 
present invention. The system includes a common object surface 1, and first and second imaging optical 
systems 102 and 202 having equivalent specifications, and normally comprising zoom lenses. The system 
also includes image sensors 103 and 203 similarly having equivalent specifications. Each of the image 
45 sensors 103 and 203 comprises a camera tube such as a saticon or a solid state imaging element such as 
a CCD. (Although Fig. 31 illustrates a single-sensor type (or single-tube type) image sensor for the sake of 
simplicity, versatility will not be disturbed even when a two-sensor type (two-tube type) image sensor or 
three-sensor type (three-tube type) image sensor via a color separation system is used.) Optical axes 101 
and 201 of these image sensors are arranged to be inclined through about 0 symmetrically about a normal 
so 0-0' to the object surface 1 so as to satisfy a condition that image fields to be picked up by these sensors 
overlap each other by a predetermined amount. Note that 20 will be referred to as the convergence angle, 
and is changed according to the image formation magnifications of the imaging optical systems (the object 
distances and the focal lengths of photographing lenses). 

Fig. 32 shows the detailed arrangement of the imaging optical systems and the image sensors. In Fig. 
55 32, a support structure such as a lens barrel is not shown. 

The first and second imaging optical systems 102 and 202 are respectively constituted by lens groups 
102a, 102b, 102c, and 102d, and lens groups 202a, 202b, 202c, and 202d. In particular, the lens groups 
102b and 202b are zoom lens groups, and the lens groups 102d and 202d are focus lens groups. The zoom 
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lens groups 102b and 202b are respectively driven by drive systems (zoom motors) 106 and 206, and 
similarly, the focus lens groups 102d and 202d are respectively driven by drive systems (focus motors) 107 
and 207. Encoders (zoom encoders) 108 and 208 are provided to the zoom lens groups 102b and 202b of 
the imaging optical systems so as to obtain position information, in the optical axis direction, of the 
5 corresponding lens groups. Focal lengths f of the imaging optical systems 102 and 202 can be obtained 
based on signals output from the encoders 108 and 208. Note that the lens groups 102d and 202d also 
have a function of compensating for image surface movement upon movement of the zoom lens groups in 
addition to the focus function. Since the arrangement of the zoom lens is well known, a detailed description 
thereof will be omitted. 

70 Encoders (focus encoders) 109 and 209 are provided to the focus lens groups 102d and 202d of the 

imaging optical systems so as to obtain position information, in the optical axis direction, of the correspond- 
ing lens groups. These encoders may comprise, e.g., externally attached members such as potentiometers, 
or may comprise systems such as pulse motors for detecting the position information, in the optical axis 
direction, of the lens by the drive systems themselves according to their drive method. The system of this 

75 embodiment further includes mechanism systems (not shown) for integrally rotating the optical system 102 
and the image sensor 103, and the optical system 202 and the image sensor 203 in opposite directions by 
the same amount in planes respectively including optical axes 101 and 201, and drive systems 
(convergence angle motors) 104 and 204 for attaining such rotations. The system also includes rotational 
angle information detecting means 105 and 205. These means may comprise, e.g., externally attached 

20 members such as rotary encoders, or may comprise systems such as pulse motors for detecting the 
position information, in the optical axis direction, of angle information by the drive systems themselves 
according to their drive method. 

A method of obtaining mis-registration amounts on the basis of image signals will be described below 
with reference to Figs. 33 and 34. Image memories 111 and 211 temporarily store signals of the overlapping 

25 portion of image signals 110 and 210. Assume that image data are mis-registered due to a deviation in 
convergence angle. A horizontal image shift process portion 20 shifts the coordinates of image data in the 
image memory 111 in the horizontal direction by a predetermined amount X (e.g., one pixel), and writes the 
shifted image data in a image memory 112. Furthermore, control for calculating a difference 5 S between the 
contents of the image memories 112 and 211 is performed using a correlation calculation process portion 

30 21 including a subtraction calculation process circuit and the like while sequentially changing the shift 
amount X so as to obtain a shift amount Xo for minimizing the difference 5 S . At this time, the shift amount Xo 
is normally obtained by an interpolation calculation process by, e.g., approximating the relationship between 
the difference 5 S and the shift amount X by, e.g., a function. 

In order to shorten the correlation calculation time, a method using only image data at the central 

35 portion of the frame of the overlapping portion, i.e., a region which includes a point O on the object surface 
1 and whose vertical and horizontal widths are shortened is effective. Alternatively, coarse adjustment may 
be attained by commonly using the encoders 105 and 205 for outputting convergence angle information, 
and fine adjustment may be attained by the above-mentioned detection using the image signals, so as to 
further reduce the area of a region to be subjected to a correlation calculation. The correlation calculation 

40 method described here is normally called a matching method. However, the present invention is not 
particularly limited to this method, and other methods such as a gradient method may be used. 

The shift amount Xo corresponding to the minimum value indicates a relative mis-registration amount 
Ay' itself of a image output from each imaging system, which mis-registration occurs due to a deviation of 
the optical axis 101 or 201 (Fig. 31) from an ideal optical axis. Therefore, convergence angle correction 

45 control is performed using a correction signal 10 for canceling the mis-registration amount Ay* of the image. 
The method of obtaining the mis-registration amount will be described below. 

Assume that an imaging operation is performed while the convergence angle has a deviation from a 
predetermined value, i.e., the optical axis 101 or 201 has an angular deviation A01 or A02 with respect to an 
axis having a predetermined angle 0. Under this assumption, the relationship between the angular deviation 

so Afli or A02 and the mis-registration amount Ay 1 of a image will be described below. For the sake of 
simplicity, a mis-registration amount Ay' of a image output from each imaging system, which amount is 
generated since the optical axis 101 is deviated to an axis 101 \ as shown in Fig. 34, will be described 
below. The mis-registration amount with respect to the optical axis 201 can be obtained by the same 
equations, as is obvious to those who are skilled in the art from the arrangement of the optical systems, and 

55 even when both the optical axes 101 and 201 have deviations, the mis-registration amount can be obtained 
as a sum of the two deviations. Therefore, a method of obtaining the mis-registration amount with respect to 
one optical axis will be described below, and a description of the method for the other axis will be omitted. 
Note that reference symbols shown in Fig. 34 are used. Furthermore, under an assumption that A01 is very 
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small, geometrically, we have: 

SotanA01 = Ay coso (1) 

5 where Ay is the deviation (PP' in Fig. 34), generated in the object surface, of an intersection P of the optical 
axis 101 and the object surface 1 from a predetermined value, and So is the distance from the front-side 
principal point of the imaging optical system (lens) 102 to an object (similarly, So' is the distance from the 
rear-side principal point to the image surface). If the image formation magnification of the imaging optical 
system 102 is represented by 0, the magnification 0 can be expressed from its definition by: 

10 

0 = AyVAy (2) 

If the focal length of the imaging system (lens) is represented by f, we have: 
75 So = 1(fi - 1)/j8 (3) 

From equations (1), (2), and (3), the mis-registration amount Ay' is given by: 
Ay* = tanA01»f<(0 - 1)/cos0 (4) 

20 

The zoom lens group 102b of the imaging optical system shown in Fig. 32 is provided with the encoder 
(zoom encoder) 108 for obtaining position information, in the optical axis direction, of the lens group, and 
the focal length f of the imaging optical system 102 can be obtained based on a signal output from the 
encoder 108. Similarly, the focus lens group 102d of the imaging optical system is provided with the 

25 encoder (focus encoder) 109 for obtaining position information, in the optical axis direction, of the lens 
group, and the image formation magnification £ of the imaging optical system 102 can be obtained based 
on both the signals from the zoom encoder 108 and the focus encoder 109. Assume that the focal lengths f 
and the image formation magnifications 0 of the two imaging optical systems 102 and 202 are caused to 
always coincide with each other by separately controlling the drive systems 105, 107, 206, and 207 on the 

30 basis of the signals from the encoders 108, 109, 208, and 209. As described above, the mis-registration 
amount Ay' of a image is obtained by a image signal process system, and the convergence angle 20, the 
focal length f, and the image formation magnification 0 are obtained by the encoders provided to each 
mechanism system. Based on these signals, a calculation control portion 22 obtains all the parameters 
except for A01 in equation (4). 

35 The calculated angular deviations A01 and A02 are supplied to a convergence angle control portion 23. 
Note that the convergence angle motors 104 and 204 have sufficient response performance for angular 
precision necessary for correcting Ay* to be equal to or smaller than a predetermined value (allowable 
value) AyO'. The control portion 23 supplies A01 to the convergence angle motor 104 as a correction signal, 
and supplies A02 to the convergence angle motor 204 as a direct control target value. Alternatively, the 

40 sum of A01 and A02 may be supplied to one of the convergence angle motors 104 and 204. Fig. 35 shows 
the flow of these signals. 

In this manner, after the mis-registration amounts of the image signals 110 and 210 output from the 
corresponding imaging systems are mechanically corrected to be equal to or smaller than the predeter- 
mined value (allowable value) AyO', images of the overlapping portion are subjected to a joint process, and 
45 thereafter, the image signals 110 can be synthesized to a single image signal 11 using, e.g., the image 
memories 111 and 211, an addition process circuit 24, and the like. Fig. 36 shows the flow of these signals. 
When such processes are sequentially performed as needed, a good panoramic image can always be 
obtained. 

A panoramic image is generally an image corresponding to a short focal length. However, as can be 
50 understood from equation (4), when the focal length f of the imaging optical system is large, or when the 
allowable value of the mis-registration amount of a image is small, the correction signals A01 and A02 also 
assume small values (on the order of several seconds), and it is difficult to obtain angular response 
performance necessary for correcting the mis-registration under the influence of, e.g., non-sensitive bands 
of the convergence angle motors 104 and 204. In this case, a method of correcting the mis-registration 
55 amount of a image output from each imaging system by a means other than the convergence angle motor 
104 or 204 is effective. Such an embodiment will be described below. The basic arrangement of the 
embodiment shown in Fig. 37 is substantially the same as that of the embodiment shown in Fig. 32, except 
that precision correction means 103a and 203a are added. 
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The first and second imaging optical systems 102 and 202 are respectively constituted by lens groups 
102a, 102b t 102c, and 102d, and lens groups 202a, 202b, 202c, and 202d. In particular, the lens groups 
102b and 202b are zoom lens groups, and the lens groups 102d and 202d are focus lens groups. The zoom 
lens groups 102b and 202b are respectively driven by drive systems (zoom motors) 106 and. 206, and 
5 similarly, the focus lens groups 102d and 202d are respectively driven by drive systems (focus motors) 107 
and 207. Encoders (zoom encoders) 108 and 208 are provided to the zoom lens groups 102b and 202b of 
the imaging optical systems so as to obtain position information, in the optical axis direction, of the 
corresponding lens groups. Similarly, encoders (focus encoders) 109 and 209 are provided to the focus lens 
groups 102d and 202d of the imaging optical systems so as to obtain position information, in the optical 

10 axis direction, of the corresponding lens groups. These encoders may comprise, e.g., externally attached 
members such as potentiometers, or may comprise systems such as pulse motors for detecting the 
position information, in the optical axis direction, of the lens by the drive systems themselves according to 
their drive method. The system of this embodiment further includes mechanism systems (not shown) for 
integrally rotating the optical system 102 and the image sensor 103, and the optical system 202 and the 

is image sensor 203 in opposite directions by the same amount in planes respectively including optical axes 
101 and 201, and drive systems (convergence angle motors) 104 and 204 for attaining such rotations. The 
system also includes rotational angle information detecting means 105 and 205. These means may 
comprise, e.g., externally attached members such as rotary encoders, or may comprise systems such as 
pulse motors for detecting the position information, in the optical axis direction, of angle information by the 

20 drive systems themselves according to their drive method. However, since the above-mentioned encoders 
are used for the auxiliary purpose, they are not always necessary in this embodiment. 

As has been described above with reference to Fig. 33, image memories 111 and 211 temporarily store 
signals of the overlapping portion of image signals 110 and 210. A horizontal image shift process portion 20 
shifts the coordinates of image data in the image memory 111 in the horizontal direction by a predeter- 

25 mined amount X (e.g., one pixel), and writes the shifted image data in a image memory 112. Furthermore, 
control for calculating a difference 5 S between the contents of the image memories 112 and 211 is 
performed using a correlation calculation process portion 21 including a subtraction calculation process 
circuit and the like while sequentially changing the shift amount X so as to obtain a shift amount Xo for 
minimizing the difference 6 S . At this time, the shift amount Xo may be obtained by an interpolation 

30 calculation process by, e.g., approximating the relationship between the difference 5 S and the shift amount X 
by, e.g., a function. When a swing motion in the vertical direction occurs as well due to insufficient precision 
of the mechanism system for driving the convergence angle motors, the same process as that for the 
horizontal direction may be performed for the vertical direction, and a two-dimensional (plane) mis- 
registration amount can be obtained. Note that a mis-registration amount Ay' (vertical) of a image in the 

35 vertical direction is given by: 

Ay' (vertical) = tanA01^f(0 - 1) (5) 

In order to shorten the correlation calculation time, a method using only image data at the central portion of 

40 the frame of the overlapping portion, i.e., a region which includes a point O on the object surface 1 and 
whose vertical and horizontal widths are shortened is effective. Alternatively, coarse adjustment may be 
attained by commonly using the encoders 105 and 205 for outputting convergence angle information, and 
fine adjustment may be attained by the above-mentioned detection using the image signals, so as to further 
reduce the area of a region to be subjected to a correlation calculation. The correlation calculation method 

45 described here is normally called a matching method. However, the present invention is not particularly 
limited to this method, and other methods such as a gradient method may be used. The shift amount Xo 
corresponding to the minimum value indicates a relative mis-registration amount Ay' itself of a image output 
from each imaging system, which mis-registration occurs due to a deviation of the optical axis 101 or 201. 
Therefore, in this embodiment, correction control of the positions in a plane of the sensors is performed 

so using a correction signal 10 according to the mis-registration amount Ay* of a image. More specifically, the 
image sensors 103 and 203 are provided with drive systems 103a and 203a capable of shifting the sensors 
in the horizontal and vertical directions. Signals calculated by a calculation control portion 22, which 
received various signals, are received by a sensor position control portion 25, and correction drive control is 
executed using the drive systems 103a and 203a. Each of the drive systems 103a and 203a comprises, 

55 e.g., a piezo element or a piezoelectric bimorph. Fig. 38 shows the flow of these signals. 

After the mis-registration amounts of the image signals 110 and 210 output from the imaging systems 
are corrected to be equal to or smaller than AyO* as a predetermined allowable value, two images can be 
synthesized using, e.g., image memories and an addition process circuit like in the above embodiment. 
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Since the flow of signals in this synthesizing process portion is common to that shown in Fig. 36, its 
illustration will be omitted here. 

The movable ranges of the drive systems 103a and 203a may be defined as follows. That is, in order to 
prevent a region to be subjected to aberration correction of the imaging optical system from being 

5 increased very much, and to prevent the load on the specifications of the drive system from being 
increased, the convergence angle encoders 105 and 205 may also be used, and the correction amount may 
be classified into coarse and fine levels with reference to a predetermined level, so that coarse adjustment 
is attained by driving the convergence angle motors 104 and 204, and fine adjustment is attained by driving 
the drive systems 103a and 203a. 

10 Fig. 39 shows stilt another embodiment of the present invention. In this embodiment, the image sensors 

103 and 203 described in the embodiment shown in Fig. 37 are specified to camera tubes, and systems for 
driving the image senrors in the horizontal or vertical direction are not used. The arrangement except for 
the drive systems 103o and 203a and the method of obtaining the correction signal 10 are common to those 
in the above embodiment, and a detailed description thereof will be omitted. 

75 In general, the camera tubes read image signals by scanning electron beams. Therefore, camera tube 
electron beam scanning locus control drivers 103b and 203b for controlling the scanning loci of electron 
beams upon reading of the image signals are provided. As described above, signals calculated by the 
calculation control portion 22 which received various signals are received by an electron beam scanning 
locus control portion 26. and correction control for shifting the scanning loci of electron beams using the 

20 drivers 103b and 203b is executed. With this control, images defined by the image signals 110 and 210 
output from the image sensors 103 and 203 can be registered. Fig. 40 shows the flow of these signals. Note 
that the above-mentioned technique for registering images by changing the scanning loci of electron beams 
is a general technique as DRC (Digital Registration Correction) in, e.g., a three-tube color television system. 
After the mis-registration amounts of the image signals 110 and 210 output from the imaging systems are 

25 corrected to be equal to or smaller than a predetermined value (allowable value) Ay0\ two images can be 
synthesized using, e.g., image memories and an addition process circuit like in the above embodiment 
(since the flow of signals in this synthesizing process portion is common to that shown in Fig. 36, its 
illustration will be omitted here). 

Fig. 41 shows still another embodiment of the present invention, wherein the image sensors comprise 

30 solid-state imaging device in place of the camera tubes, and systems for driving the sensors in the 
horizontal or vertical direction are not used. More specifically, in this embodiment, the image sensors 103 
and 203 described in the embodiment shown in Fig. 39 specifically use solid-state imaging device. 
Therefore, the basic arrangement and the method of obtaining the correction signal 10 are common to those 
in the embodiments shown in Figs. 37 and 39, and a detailed description thereof will be omitted. 

35 As shown in Fig. 42, signals calculated by a calculation control portion 22, which received various 
signals, are received by a image position control portion 27. Image memories 111 and 211 temporarily store 
image signals 110 and 210. Images defined by image data in this stage are not registered due to 
convergence angle errors. A correction conversion portion 28 converts the coordinates of image, data. More 
specifically, the portion 28 executes control for shifting image data in the memory 111 by the calculated 

40 mis-registration amount Ay* of a image, and writing the shifted image data in a image memory 112. 
Although the image memories 112 and 111 are independently arranged for the sake of simplicity, a 
common memory may be used in place of these memories. When the coordinates of image data are 
shifted, a data interpolation calculation process may be performed as needed. In this embodiment, the 
coordinates of image data in the image memory 111 are shifted. Of course, the coordinates of image data in 

45 the image memory 211 may be shifted. Then, two images can be synthesized based on image data in the 
image memories 112 and 211, as shown in Fig. 36, using, e.g., the image memories and an addition 
process circuit like in the above embodiment 

In the above embodiment, correction control of the mis-registration amounts Ay' of images due to 
convergence angle errors is executed. In some cases, mis-registration amounts of images due to a 

50 difference between image formation magnifications of imaging optical systems must also be taken into 
consideration. 

Such mis-registration amounts can also be detected using image signals independently of the encoders 
108 and 109 for obtaining position information, in the optical axis direction, of the lenses. Such an 
embodiment will be described below. 
55 Fig. 43 shows still another embodiment of the present invention. The basic arrangement of this 
embodiment is the same as that of the embodiment shown in Fig. 37, and since the same reference 
numerals in Fig. 43 denote the same parts as in Fig. 37, a detailed description thereof will be omitted. The 
image memories 111 and 211 temporarily store image signals of the overlapping portions of the image 
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signals 110 and 210. In this stage, images defined by image data are not registered due to a difference 
between the image formation magnifications of the imaging optical systems 102 and 202. In this case, 
assume that mis-registration amounts (Ay* described above) of the central positions of two images have 
already been detected by, e.g., the method described in the embodiments shown in Figs. 32 to 42, and the 

5 central positions of the two images have been corrected. 

As one method of a magnification correlation calculation process, the following method is known. More 
specifically, a magnification calculation process portion 29 converts the coordinates of image data in the 
image memory 111 by multiplying the coordinates with a constant k proportional to the distance from the 
center of the frame, and writes the coordinate-converted image data in a image memory 112. A difference 

10 5 m between the contents of the image memories 112 and 21 1 is then calculated using a subtraction process 
circuit 30. The above-mentioned subtraction process is repeated while sequentially changing the constant k, 
thereby obtaining a constant ko corresponding to the minimum difference 5 m . Fig. 44 shows the flow of 
these signals. The constant ko represents the ratio of the image formation magnification of the imaging 
optical system 102 to that of the imaging optical system 202, and is defined as a correction signal 12. 

75 Fig. 45 shows a method of shortening the above-mentioned magnification correlation process time. 

In Fig. 45, only image signal portions 110a1 and 110a2, and 210a1 and 210a2 corresponding to two 
upper and lower regions 111A1 and 111 A2 and two upper and lower regions 211A1 and 211A2, which 
portions respectively include a line Y-Y' passing through a point 0 on an object surface 1 and perpendicular 
to a plane including the optical axes 101 and 201, and are separated from the central portion 0, are used. 

20 Representative points of these regions on the object surface are represented by Q and R, and images of 
these points are represented by Q1\ R1\ Q2', and R2\ The positional deviation correlation, in the vertical 
direction of these points may be calculated by, e.g., the method described in the embodiments shown in 
Figs. 32 to 42, and the ratio of a length QV-RV to a length Q2'-R2* may be calculated to obtain the 
constant ko. This technique utilizes the fact that the detection sensitivity of the difference between the 

25 image formation magnifications can be improved toward the periphery of the frame, and the fact that the 
difference between the image formation magnifications can be regarded as a positional deviation in a small 
region. The constant ko is obtained, as described above, and the following correction control can be 
executed based on this constant. 

More specifically, as a mis-registration correction method, when the zoom motors 106 and 206 have 

30 response performance necessary for obtaining a constant ko for correcting the mis-registration amounts of 
images caused by the difference between the image formation magnifications to be equal to or smaller than 
a predetermined value (allowable value), servo control is executed by supplying a correction signal to one 
or both of the zoom motors 106 and 206 so as to compensate for the magnifications. 

When it is difficult for the zoom motors 106 and 206 to obtain response performance necessary for 

35 correcting the mis-registration amounts due to the influence of the non-sensitive bands, correction control 
by changing the scanning loci of the electron beams described above with reference to Fig. 39 or correction 
control by converting the coordinates of the image memory in the above embodiment may be suitably 
executed. In this case, upon coordinate conversion of the image memory, an interpolation calculation 
process of image data may be performed as needed. 

40 As described above, an imaging apparatus which can obtain a panoramic image in which the boundary 
line is not conspicuous can be provided. 

In the above embodiment, a relatively large aspect ratio is set. However, an image having a desired 
aspect ratio can be obtained by changing the convergence angle 20. Fig. 46 shows the principle of a 
synthesizing system. In Fig. 46, an image having a new aspect ratio is obtained by synthesizing images 

45 from right and left side imaging systems. 

In this case, although the basic arrangement is the same as that shown in Fig. 31, the convergence 
angle 2d is selected to be smaller than that of a panoramic image. 

More specifically, the optical axes 101 and 201 shown in Fig. 31 are set to be symmetrically inclined 
through 6 with respect to the normal O-O' to the object surface 1 , so that two image fields overlap each 

50 other by a predetermined amount according to a selected aspect ratio of a frame. 

For example, Fig. 47 shows a mode for converting an aspect ratio of 4 : 3 of images picked up by 
NTSC image sensors into an aspect ratio of 16 : 9 of an HD system. 

Therefore, a desired aspect ratio is determined first, and the convergence angle corresponding to the 
determined aspect ratio is set by a setting means 115 (Fig. 48) by driving the convergence angle motors 

55 104 and 204. Thereafter, the registration compensation method described in each of the above embodi- 
ments is applied to obtain a desired image. 

The descriptions of the above embodiments have been made under an assumption that an object is a 
substantially flat one or can be regarded as a flat object. Next, an embodiment that can be applied to an 

24 

3NSDCCID: <EP 0563737A1_!_> 



EP 0 563 737 A1 

object whose depth cannot be ignored will be described below. 

Fig. 49 shows a basic arrangement of a multi-lens imaging system according to still another embodi- 
ment of the present invention. The system shown in Fig. 49 includes a common object surface 1, and first 
and second imaging optical systems 102 and 202 having equivalent specifications, and normally comprising 

5 zoom lenses, as will be described later. Image sensors 103 and 203 similarly have equivalent specifications, 
and comprise camera tubes such as saticons, or solid-state imaging device such as CCDs. Although Fig. 49 
illustrates a single-sensor type (or single-tube type) image sensor for the sake of simplicity, versatility will 
not be disturbed even when a two-sensor type (two-tube type) image sensor or three-sensor type (three- 
tube type) image sensor via a color separation system is used. 

io Optical axes 101 and 201 of these imaging optical systems are set to cross at a point O on the object 

surface 1 and to be symmetrically inclined through Q with respect to a normal O-O' to the object surface 1 
while passing the point O. Note that 29 is defined as the convergence angle, and imaging operations are 
performed by changing the convergence angle according to a change in object distance S. 

Fig. 50 shows the detailed arrangement of the imaging optical systems 102 and 202, and Fig. 51 is a 

75 block diagram showing functions of the members constituting the imaging optical systems. The first and 
second imaging optical systems 102 and 202 are respectively constituted by lens groups 102a, 102b, 102c, 
and 102d, and lens groups 202a, 202b, 202c, and 202d. In particular, the lens groups 102b and 202b are 
zoom lens groups, and the lens groups 102d and 202d are focus lens groups. The zoom lens groups 102b 
and 202b are respectively driven by drive systems (zoom motors) 106 and 206, and similarly, the focus lens 

20 groups 102d and 202d are respectively driven by drive systems (focus motors) 107 and 207. The system of 
this embodiment further includes mechanism systems (not shown) for integrally rotating the optical system 
102 and the image sensor 103, and the optical system 202 and the image sensor 203 in opposite directions 
by the same amount in planes respectively including optical axes 101 and 201, and drive systems 
(convergence angle motors) 104 and 204 for attaining such rotations. 

25 Angle encoders (rotational angle detecting means) 105 and 205 respectively measure the rotational 
angles of the imaging optical systems 102 and 202. Zoom encoders 108 and 208 respectively measure the 
moving amounts of the zoom lens groups 102b and 202b to obtain zoom ratios. Focus encoders 109 and 
209 measure the positions of the focus lens groups. Image signals 110 and 210 are output from the image 
sensors 1 03 and 203, and are stored in image memories 1 1 1 and 21 1 . 

30 Functions of a calculation control portion 12, a correlation calculation portion 13, and a correction 
calculation portion 14 will be described later. 

Detection of position information of an object when imaging operations are performed in an arrangement 
shown in Fig. 52 will be described below. 

As shown in Fig. 52, x-, y- and z-axes are defined to have the above-mentioned point O on the object 

35 surface as an origin. 

The intersections between the imaging optical systems 102 and 202 and their optical axes 101 and 201 
are represented by Q R and Q L , the distance from the front-side principal point of the imaging optical system 
(lens) 102 or 202 to the above-mentioned point O is represented by So, and the distance from the rear-side 
principal point to the image sensor 103 or 203 is represented by So 1 - A method of obtaining coordinates P2 
40 (xc, zo) on the object surface 1 in the x-z plane shown in Fig. 53 will be briefly described below. 

If the intersections between the imaging optical systems 102 and 202 and their optical axes 101 and 
201 are respectively represented by QR (xi, -zi) and Q L (-xi , -zi), the coordinates x, and zi can be 
geometrically expressed as follows using the imaging condition So and the convergence angle 8: 

45 xi = Sosine (1-a) 

zi = Socose (1-b) 

Similarly, as for intersections Or' (xz, -Z2) and 0 L f (-X2, -Z2) between the image sensors 103 and 203 
50 and the optical axes 101 and 201, their coordinates are expressed by: 

x 2 = (So + So')sinG (2-a) 

Z2 = (So + So')cose (2-b) 

55 

As shown in Fig. 53, if image points on the image sensors 103 and 203 with respect to an object point 
P2 (xo, zo) on an object are respectively represented by P R * (x r , -z r ) and Pi (-x L , -z L ), and the image 
heights on the image sensors 103 and 203 are respectively represented by X R ' and X L \ geometrically, we 
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have: 

x R = (So + S o ')sin0 - x R 'cos0 (3-a) 

5 z R = (So + So')cos0 + x R 'sin0 (3-b) 

x L = (So + So')sin6 + x L 'cos0 (3-c) 

z L = (So + Sj')cos9 - x L *sin9 (3-d) 

At this time, if a line passing the points P R and Q R is represented by f R (So, S c \ 0, x R '), and a line 
passing the points P L * and Q L is represented by f L (So, So', 6, x L '), the point P 2 (xo, zo) on the object surface 
corresponds to the coordinates of the intersection of these two lines according to the definition. 

Also, as shown , in Fig. 52, y 3 can be expressed as follows in association with the imaging optical 
system 102: 

yo = YrSrVSr (4) 

S R is the distance from the object point Pr (x 0 , zo) to the front-side principal point of the imaging optical 
20 system 102 in Fig. 53, and S n ' is the distance from the rear-side principal point of the imaging optical 
system 102 to the image pint P n ' on the image sensor 103 in Fig. 53. 
At this time, a point P (x 0 . yo. z 0 ) in Fig 52 is defined by: 

p = f(So. So\ 6, x R \ x L \ Y R(L /) (5) 

25 

That is, the point P is expressed by a function of imaging conditions (So, So', 0) and an output image (x R \ 
xl\ y r(L)*) of each image sensor, and position information on the object surface can be obtained by 
detecting these parameters. 

The method of obtaining parameters in the above-mentioned process will be described below with 

30 reference to Figs. 50 and 51. First, the convergence angle 2e is detected by the rotational angle detecting 
means 105 and 205 such as rotary encoders. The encoders (zoom encoders) 108 and 208 for obtaining 
position information, in the optical axis direction, of the corresponding lens groups are used, and the focal 
lengths f of the imaging optical systems 102 and 202 are obtained based on the signals output from these 
encoders. Similarly, the encoders (focus encoders) 109 and 209 for obtaining position information, in the 

35 optical axis direction, of the corresponding lens groups are provided to the focus lens groups 102d and 
202d of the imaging optical systems. These encoders may comprise, e.g., externally attached members 
such as potentiometers, or may comprise systems such as pulse motors for detecting the position 
information, in the optical axis direction, of the lens by the drive systems themselves according to their 
drive method. The calculation control portion 12 calculates the object distances So to the imaging optical 

40 systems 102 and 202 on the basis of the signals from the focus encoders 109 and 209. Furthermore, the 
portion 12 calculates lens backs So' of the imaging optical systems 102 and 202 together with the focal 
lengths f of the imaging optical systems 102 and 202. Note that the focal lengths f and the lens backs So* of 
the two imaging optical systems 102 and 202 are caused to always coincide with each other by 
independently controlling the drive systems 106, 107, 206, and 207 on the basis of the signals from the 

45 encoders 108, 109, 208, and 209. 

As described above, the calculation control portion 12 calculates So (the distance from the front-side 
principal point of the imaging optical system (lens) to the intersection of the corresponding optical axis), So* 
(the distance from the rear-side principal point of the imaging optical system (lens) to the image surface), 
and 0 (convergence angle) on the basis of the signals from the encoders 105, 108, 109, 205, 208, and 209. 

so The image memories 111 and 211 temporarily store image signals 110 and 210. 

The correlation calculation process portion 13 performs a correlation calculation about image data 
stored in the image memories 111 and 211. Figs. 54A and 54B show the principle of a correlation 
calculation process. In the process, as shown in Fig. 54B, a pixel data group 112 having pixel data at a 
point (x iR \ y JR ') as a center is defined as one block, and a correlation calculation is performed between this 

55 block and a corresponding image block on the image memory 211. Fig. 55 illustrates the relationship 
between a correlation value and an x' (y*) axis coordinate in the correlation calculation in the horizontal and 
vertical directions. In Fig. 55, X L ' (y L ') corresponding to a maximum correlation value is obtained with a 
precision equal to or lower than the sampling pitch of the image sensor by, e.g., approximating the 
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relationship near the correlation peak by a function. The block 112 is also defined for other pixel data Rjj on 
the image memory 111, and a similar calculation process is executed to obtain X L * and y L ' for each pixel 
data. 

The coordinates P R (xo, yo, z 0 ) on the object surface corresponding to each pixel data Ry on the image 
5 sensor 103 can be calculated from the relationship given by equation (5) using the image information (x iR \ 
yj*') obtained by these series of processes, and the imaging conditions (So, So*, 0)- 
The same calculation process is executed for each pixel data Ly (x iL \ Y jL ') of the image sensor 203 to 
obtain x R ' and y R ' for each pixel, and the coordinates P L (xo, yo, zo) on the object surface corresponding to 
each pixel data on the image sensor 203 are then calculated. Note that the coordinates Pr and P L on the 
70 object surface do not always coincide with each other. 

The correction process portion 14 executes a process on the basis of the coordinates P R and P L 
obtained by the above-mentioned process. In the correction process portion 14, a desired view position is 
input. At th«s time, when the view position is input, as shown in Fig. 56, image points (x\ y') corresponding 
to the coordinate values can be obtained as follows using the above-mentioned imaging conditions So and 
75 So' and the coordinates P R(L ) (xo, yo, zo) on the object surface: 

x' = xoSc"(So + zo) . (6-a) 

y' = y 0 SoV(So + zo) (6-b) 

20 

The coordinates (x iRa) \ Y in(L) ') on each image memory are coordinate-converted by a calculation process on 
the basis of equations (6-a) and (6-b), and the converted data are written in a image memory 15. The mis- 
registration amounts have been removed from the image data stored in the image memory 15. and the 
image data is ideally twice image data output from the image sensors 103 and 203. As a result, a high-fine 

25 output image can be obtained. 

The embodiment described above with reference to Fig. 49 and subsequent drawings can be applied to 
a panoramic image and images having various aspect ratios. In this case, since an operation for extracting 
corresponding points on at least partial regions of a plurality of images is required, the images must 
partially overlap each other. A plurality of imaging optical systems of every embodiment does not always lie 

30 in the same mode. For a different purpose, a different mode imaging optical system is sometimes used. 

With the above-mentioned arrangements of the present invention, the following effects can be expected. 
Even when images obtained by imaging systems suffer from keystone distortions, the keystone 
distortions can be removed, and the images can be converted into desired images, thus obtaining a high- 
fine image free from keystone distortions. 

35 Since at least one of a mechanism system or image process system is controlled so as to correct a 
mis-registration amount detected by a detecting means to be equal to or smaller than a predetermined 
value, images obtained by imaging systems can be registered and synthesized. For this reason, an 
apparatus which can obtain a high-fine image by picking up an image of a common object using a plurality 
of imaging systems can be provided. 

40 Also, an apparatus capable of obtaining an image having very high image quality and a desired aspect 
ratio, and an apparatus capable of obtaining a panoramic image free from a conspicuous boundary line can 
be provided. 

When a multi-lens imaging apparatus of the present invention is connected to a display device such as 
a CRT via an image process apparatus, it can be utilized as a high-performance panoramic TV system, a 

45 3D TV system, an HDTV system, and the like. Similarly, when the apparatus of the present invention is 
connected to a recording apparatus via the image process apparatus, it can be utilized as a high- 
performance recording system. 

This invention relates to a multi-lens imaging apparatus and, more particularly, to a multi-lens imaging 
apparatus for obtaining a single high-fine image by synthesizing a plurality of images obtained by picking 

50 up an image of a common object using a plurality of imaging systems and to an apparatus which can 
provide an image having an aspect ratio different from that of an image obtained by a single imaging 
apparatus, especially, a panoramic image, by synthesizing a plurality of images using a plurality of imaging 
systems. This invention has as its object to obtain a high-fine image by synthesizing images from a plurality 
of imaging systems and, more specifically, to correct mis-registration such as keystone distortions 

55 generated in the images. 
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Claims 

1. A multi-lens imaging apparatus comprising: 

a plurality of imaging optical systems; 
5 image sensing means for sensing images of an object corresponding to said plurality of imaging 

optical systems via said plurality of imaging optical systems; 

detection means for detecting displacement amounts of registration amounts of the images of the 
object obtained from said plurality of imaging optical systems; 

correction means for correcting the displacement amounts to be less than a predetermined value 
10 according to a detection signal from said detection means; and 

synthesizing means for synthesizing the images of the object. 

2. An apparatus according to claim 1 , wherein said detection means comprises: 

convergence angle detection means for detecting an angle defined between optical axes of said 
75 plurality of imaging optical systems; 

zoom encoders for detecting focal lengths of said imaging optical systems; and 
focus encoders for detecting object distances to said imaging optical systems. 

3. An apparatus according to claim 2, wherein said image sensing means comprise camera tubes, and 
20 said apparatus further comprises correction signal generation means for generating a correction signal 

of the displacement amounts of registration on the basis of output signals from said convergence angle 
detection means, said zoom encoders, and said focus encoders, and camera tube electron beam 
scanning locus control drivers for controlling electron beams of said camera tubes on the basis of the 
correction signal. 

25 

4. An apparatus according to claim 2, wherein said image sensing means comprise solid-state image 
pickup elements, and said apparatus further comprises image memories for storing a plurality of image 
data from said solid-state image pickup elements, correction conversion means for obtaining the 
displacement amounts of registration of the plurality of image data, and executing coordinate conver- 

30 sion of the image data to cancel the displacement amounts of registration, and synthesizing process 
means for synthesizing the plurality of image data. 

5. An apparatus according to claim 1 , wherein optical axes of said plurality of image pickup optical 
systems are not parallel to each other, and the said displacements of registration are caused by 

35 keystone distortions of the images of the object. 

6. An apparatus according to claim 1, wherein the displacements of registration are caused by a 
difference between image formation magnifications of said plurality of imaging optical systems, and 
said apparatus comprises a magnification correlation calculation process portion and a calculation 

40 process portion. 

7. An apparatus according to claim 6, wherein said magnification correlation calculation process portion 
calculates positional deviations associated with distances from a center to corresponding positions on a 
plurality of frames, and inputs a constant for correcting the positional deviations to said calculation 

45 process portion. 

8. An apparatus according to claim 7, wherein said magnification correlation calculation process portion 
executes a correlation calculation using image signals in a partial region having a line passing central 
portions of the plurality of frames and perpendicular to a plane including the optical axes of at least two 

50 of said imaging optical systems defined as a center. 

9. A multi-lens imaging apparatus comprising: 

a plurality of imaging optical systems whose optical axes are not parallel to each other; 
image sensing means for sensing images of an object corresponding to said plurality of imaging 
55 optical systems via said plurality of imaging optical systems; 

keystone distortion correction means for correcting a keystone distortion generated in at least one 
image obtained through said image sensing means; and 

synthesizing means for synthesizing the plurality of images. 
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10. An apparatus according to claim 9, wherein said keystone distortion correction means corrects the 
keystone distortion with reference to focal lengths, image formation magnifications, and inclination 
angles of the optical axes with respect to an object surface of said plurality of imaging optical systems. 

5 11. An apparatus according to claim 9, wherein said keystone distortion correction means executes a 
conversion process of images obtained by said plurality of imaging optical systems on the basis of 
focal lengths and image formation magnifications of said plurality of imaging optical systems using an 
arbitrary conversion variable so as to obtain a difference between the converted images, and corrects 
the keystone distortion using the conversion variable to minimize the difference between the images. 

w 

12. An apparatus according to claim 9, wherein said keystone distortion correction means detects a 
distortion amount cf the keystone distortion using, as parameters which are focal lengths, image 
formation magnifications, and inclination angles of the optical axes with respect to an object surface of 
said plurality of imaging optical systems, calculates a difference between an image from which the 

75 keystone distortion is corrected according to the detected distortion amount and a desired image, and 

sets up the inclination angles to minimize the difference, thereby correcting the keystone distortion. 

13. An apparatus according to claim 9, wherein said image sensing means comprise camera tubes. 

20 14. An apparatus according to claim 9, wherein said image sensing means comprise solid-state imaging 
device. 

15. A multi-lens imaging apparatus for picking up an image using a plurality of imaging optical systems 
while image fields at least partially overlap each other, comprising: 

25 detection means for extracting image signals of the overlapping portion of images obtained from 

said plurality of imaging optical systems, and detecting a displacement amount of registration of the 
overlapping portion; and 

correction means for correcting the displacement amount of registration to be less than a 
predetermined value according to a detection signal from said detection means. 

30 

16. An apparatus according to claim 15, further comprising image sensing means for sensing images from 
said plurality of imaging optical systems. 

17. An apparatus according to claim 16, further comprising means for synthesizing a plurality of image. 
35 signals output from said image sensing means. 

18. An apparatus according to claim 15, further comprising convergence angle changing means for making 
an angle formed between optical axes of said plurality of imaging optical systems change. 

40 19. A multi-lens imaging apparatus comprising: 
a plurality of imaging optical systems; 

means for detecting imaging conditions of said plurality of imaging optical systems; 

image sensing means for sensing images of an object via said plurality of imaging optical systems; 

means for detecting position information of the object from the image signals from said image 
45 sensing means; and 

image synthesizing means for synthesizing the plurality of images on the basis of the imaging 
conditions and the position information so that corresponding points of the plurality of images coincide 
with each other. 

so 20. An apparatus according to claim 19, wherein said image synthesizing means forms a synthesized 
image with reference to a predetermined view position. 

21. A multi-lens imaging apparatus for picking up an image of a common object using a plurality of imaging 
optical systems, comprising: 
55 image sensing means for sensing images of the object corresponding to said plurality of imaging 

optical systems; 

detection means for detecting three-dimensional position information of a plurality of points in each 
of the images obtained by said image sensing means; and 
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synthesizing means for generating a synthesized image of the plurality of images on the basis of 
the position information. 

22. An image recording system comprising: 
5 a multi-lens imaging apparatus; and 

a recording apparatus for executing an image process of an image signal from said multi-lens 
imaging apparatus, and recording the processed image signal, wherein 

said multi-lens imaging apparatus includes: 

a plurality of imaging optical systems; 
w image sensing means for sensing images of an object corresponding to said plurality of imaging 

optical systems via said plurality of imaging optical systems; 

detection moans for detecting displacement amounts of registration of the images of the object 
obtained from said plurality of imaging optical systems; 

correction means for correcting the displacement amounts of registration to be less than a 
75 predetermined value according to a detection signal from said detection means; and 

synthesizing means for synthesizing the images of the object. 

23. A TV system comprising: 

a multi-lens imaging apparatus; 
20 an image process apparatus for synthesizing a plurality of image signals from said multi-lens 

imaging apparatus; and 

a display apparatus for displaying an image according to an image signal from said image process 
apparatus, 

said multi-lens imaging apparatus comprising: 
25 a plurality of imaging optical systems; 

image sensing means for sensing images of an object corresponding to said plurality of imaging 
optical systems via said plurality of imaging optical systems; 

detection means for detecting mis-registration amounts of the images of the object obtained from 
said plurality of imaging optical systems; 
30 correction means for correcting the displacement amounts of registration to be less than a 

predetermined value according to a detection signal from said detection means; and 
synthesizing means for synthesizing the images of the object 
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